2023-12-16T20:32:42.673 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-16T20:32:42.674 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494793 --owner scheduled_yuriw@teuthology --name yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494793/orig.config.yaml 2023-12-16T20:32:42.726 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-16T20:32:42.811 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494793 branch: wip-yuri8-testing-2023-12-15-0911 description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7494793' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri8-testing-2023-12-15-0911 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2023-12-15-0911 sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 2960 sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri8-testing-2023-12-15-0911 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 targets: smithi052.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNUXelHJu4JJ7UoAMmLBbKww0hCCLDrnPDr5Hv1tiv/I80DAhvcxbJ+NCL5jHkJ5c2RJJ8kU/QoE2YAW1j0nKkw= smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAE8XanDXS8HV8HzoKZ8bTXBKiWHqinDwT6anLD6hVY9hEIaHPxyUni6WKqjgHjWB85SKNd0Re2EUyO+B5GPOrs= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-16_15:45:14 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-12-16T20:32:42.811 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa; will attempt to use it 2023-12-16T20:32:42.812 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks 2023-12-16T20:32:42.812 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-16T20:32:42.814 INFO:teuthology.task.internal:Checking packages... 2023-12-16T20:32:42.839 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c7586e95519b7be76f3b6e3a4b333888baf46bc9' 2023-12-16T20:32:42.839 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T20:32:42.839 INFO:teuthology.packaging:ref: None 2023-12-16T20:32:42.839 INFO:teuthology.packaging:tag: None 2023-12-16T20:32:42.839 INFO:teuthology.packaging:branch: wip-yuri8-testing-2023-12-15-0911 2023-12-16T20:32:42.839 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T20:32:42.839 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri8-testing-2023-12-15-0911 2023-12-16T20:32:43.065 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-202.gc7586e95 2023-12-16T20:32:43.069 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-16T20:32:43.076 INFO:teuthology.task.internal:no buildpackages task found 2023-12-16T20:32:43.076 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-16T20:32:43.116 INFO:teuthology.task.internal:Saving configuration 2023-12-16T20:32:43.137 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-16T20:32:43.143 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T20:32:43.168 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494793', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 20:27:43.670978', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNUXelHJu4JJ7UoAMmLBbKww0hCCLDrnPDr5Hv1tiv/I80DAhvcxbJ+NCL5jHkJ5c2RJJ8kU/QoE2YAW1j0nKkw='} 2023-12-16T20:32:43.192 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494793', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 20:27:43.672111', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAE8XanDXS8HV8HzoKZ8bTXBKiWHqinDwT6anLD6hVY9hEIaHPxyUni6WKqjgHjWB85SKNd0Re2EUyO+B5GPOrs='} 2023-12-16T20:32:43.193 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-16T20:32:43.207 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-12-16T20:32:43.207 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-12-16T20:32:43.207 INFO:teuthology.run_tasks:Running task console_log... 2023-12-16T20:32:43.330 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f95f355df70>, signals=[15]) 2023-12-16T20:32:43.330 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-16T20:32:43.398 INFO:teuthology.task.internal:Opening connections... 2023-12-16T20:32:43.398 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2023-12-16T20:32:43.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T20:32:43.500 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2023-12-16T20:32:43.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T20:32:43.585 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-16T20:32:43.593 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2023-12-16T20:32:43.611 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2023-12-16T20:32:43.612 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2023-12-16T20:32:43.669 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Stream" 2023-12-16T20:32:43.669 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="9" 2023-12-16T20:32:43.669 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2023-12-16T20:32:43.670 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2023-12-16T20:32:43.670 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="9" 2023-12-16T20:32:43.670 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el9" 2023-12-16T20:32:43.670 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-16T20:32:43.670 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2023-12-16T20:32:43.670 INFO:teuthology.orchestra.run.smithi052.stdout:LOGO="fedora-logo-icon" 2023-12-16T20:32:43.670 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-16T20:32:43.670 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/" 2023-12-16T20:32:43.670 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-16T20:32:43.670 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-16T20:32:43.670 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-16T20:32:43.671 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2023-12-16T20:32:43.691 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2023-12-16T20:32:43.710 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2023-12-16T20:32:43.710 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2023-12-16T20:32:43.768 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2023-12-16T20:32:43.768 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="9" 2023-12-16T20:32:43.768 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2023-12-16T20:32:43.769 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2023-12-16T20:32:43.769 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="9" 2023-12-16T20:32:43.769 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el9" 2023-12-16T20:32:43.769 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-16T20:32:43.769 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2023-12-16T20:32:43.769 INFO:teuthology.orchestra.run.smithi150.stdout:LOGO="fedora-logo-icon" 2023-12-16T20:32:43.769 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-16T20:32:43.769 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2023-12-16T20:32:43.769 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-16T20:32:43.769 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-16T20:32:43.769 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-16T20:32:43.770 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-12-16T20:32:43.807 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-16T20:32:43.816 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-16T20:32:43.853 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-16T20:32:43.854 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2023-12-16T20:32:43.856 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2023-12-16T20:32:43.872 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-16T20:32:43.878 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-16T20:32:43.878 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2023-12-16T20:32:43.913 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2023-12-16T20:32:43.944 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-16T20:32:43.999 INFO:teuthology.run_tasks:Running task kernel... 2023-12-16T20:32:44.013 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-16T20:32:44.014 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-16T20:32:44.014 DEBUG:teuthology.orchestra.run.smithi052:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-16T20:32:44.015 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-16T20:32:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T20:32:44.032 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2023-12-16T20:32:44.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T20:32:44.033 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2023-12-16T20:32:44.090 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-361.el9.x86_64 2023-12-16T20:32:44.090 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.14.0-361.el9.x86_64 2023-12-16T20:32:44.091 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2023-12-16T20:32:44.093 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-361.el9.x86_64 2023-12-16T20:32:44.093 INFO:teuthology.task.kernel:Running kernel on smithi052: 5.14.0-361.el9.x86_64 2023-12-16T20:32:44.093 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum install -y kernel 2023-12-16T20:32:46.231 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - AppStream 15 MB/s | 6.4 MB 00:00 2023-12-16T20:32:46.262 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2023-12-16T20:32:47.587 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2023-12-16T20:32:47.606 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - BaseOS 7.7 MB/s | 2.0 MB 00:00 2023-12-16T20:32:48.614 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 7.9 MB 00:00 2023-12-16T20:32:50.565 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 22 MB/s | 18 MB 00:00 2023-12-16T20:32:50.769 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 7.9 MB 00:02 2023-12-16T20:32:54.395 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 7.6 MB/s | 18 MB 00:02 2023-12-16T20:32:55.491 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 5.8 MB 00:01 2023-12-16T20:32:57.116 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2023-12-16T20:32:58.551 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2023-12-16T20:32:58.996 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 5.8 MB 00:01 2023-12-16T20:33:00.505 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2023-12-16T20:33:01.431 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 24 MB/s | 20 MB 00:00 2023-12-16T20:33:03.451 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2023-12-16T20:33:05.683 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-16T20:33:05.683 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-16T20:33:05.784 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:33:05.792 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:33:05.792 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2023-12-16T20:33:05.792 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:33:05.792 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:33:05.793 INFO:teuthology.orchestra.run.smithi052.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-16T20:33:05.793 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-12-16T20:33:05.793 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-16T20:33:05.793 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-16T20:33:05.793 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-16T20:33:05.793 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:33:05.793 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:33:05.793 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:33:05.793 INFO:teuthology.orchestra.run.smithi052.stdout:Install 4 Packages 2023-12-16T20:33:05.793 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:33:05.793 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 96 M 2023-12-16T20:33:05.793 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 121 M 2023-12-16T20:33:05.793 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:33:06.145 INFO:teuthology.orchestra.run.smithi052.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 18 MB/s | 5.6 MB 00:00 2023-12-16T20:33:06.164 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2023-12-16T20:33:06.620 INFO:teuthology.orchestra.run.smithi052.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 26 MB/s | 20 MB 00:00 2023-12-16T20:33:07.037 INFO:teuthology.orchestra.run.smithi052.stdout:(3/4): kernel-modules-core-5.14.0-391.el9.x86_6 36 MB/s | 32 MB 00:00 2023-12-16T20:33:07.554 INFO:teuthology.orchestra.run.smithi052.stdout:(4/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 22 MB/s | 38 MB 00:01 2023-12-16T20:33:07.554 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:33:07.554 INFO:teuthology.orchestra.run.smithi052.stdout:Total 55 MB/s | 96 MB 00:01 2023-12-16T20:33:08.254 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-16T20:33:08.254 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-16T20:33:08.334 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:33:08.340 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:33:08.340 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2023-12-16T20:33:08.340 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:33:08.340 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:33:08.340 INFO:teuthology.orchestra.run.smithi150.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-16T20:33:08.340 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-12-16T20:33:08.341 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-16T20:33:08.341 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-16T20:33:08.341 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-16T20:33:08.341 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:33:08.341 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:33:08.341 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:33:08.341 INFO:teuthology.orchestra.run.smithi150.stdout:Install 4 Packages 2023-12-16T20:33:08.341 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:33:08.342 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 96 M 2023-12-16T20:33:08.342 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 121 M 2023-12-16T20:33:08.342 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:33:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:33:08.690 INFO:teuthology.orchestra.run.smithi150.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 18 MB/s | 5.6 MB 00:00 2023-12-16T20:33:08.898 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:33:08.898 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:33:09.008 INFO:teuthology.orchestra.run.smithi150.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 33 MB/s | 20 MB 00:00 2023-12-16T20:33:09.424 INFO:teuthology.orchestra.run.smithi150.stdout:(3/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 36 MB/s | 38 MB 00:01 2023-12-16T20:33:09.767 INFO:teuthology.orchestra.run.smithi150.stdout:(4/4): kernel-modules-core-5.14.0-391.el9.x86_6 30 MB/s | 32 MB 00:01 2023-12-16T20:33:09.767 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:33:09.767 INFO:teuthology.orchestra.run.smithi150.stdout:Total 67 MB/s | 96 MB 00:01 2023-12-16T20:33:10.292 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:33:10.292 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:33:10.623 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:33:10.997 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:33:10.998 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:33:12.311 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:33:12.311 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:33:13.095 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:33:13.756 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-16T20:33:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T20:33:15.202 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:33:15.841 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-16T20:33:15.975 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T20:33:16.777 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T20:33:16.949 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T20:33:18.737 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T20:33:18.918 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T20:33:21.942 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T20:33:22.076 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T20:33:23.554 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T20:33:23.734 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T20:33:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T20:33:28.393 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T20:33:57.196 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T20:33:57.237 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-16T20:33:57.621 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T20:33:57.665 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-16T20:33:57.726 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T20:33:57.726 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-16T20:33:57.726 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T20:33:57.726 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T20:33:58.101 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T20:33:58.101 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-16T20:33:58.102 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T20:33:58.102 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T20:33:58.126 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T20:33:58.126 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:33:58.126 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:33:58.126 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-16T20:33:58.126 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-16T20:33:58.126 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-16T20:33:58.127 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-16T20:33:58.127 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:33:58.127 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:33:58.566 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T20:33:58.566 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:33:58.566 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:33:58.566 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-16T20:33:58.566 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-16T20:33:58.566 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-16T20:33:58.566 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-16T20:33:58.566 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:33:58.566 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:33:58.619 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T20:33:58.760 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T20:33:58.761 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T20:33:58.761 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-16T20:33:58.761 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-16T20:33:58.761 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-16T20:33:58.761 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-12-16T20:33:58.762 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=None, version=distro) 2023-12-16T20:33:58.762 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T20:33:58.879 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T20:33:58.880 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T20:33:58.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo rpm -qi grub2-tools 2023-12-16T20:33:58.920 INFO:teuthology.orchestra.run.smithi150.stdout:Name : grub2-tools 2023-12-16T20:33:58.920 INFO:teuthology.orchestra.run.smithi150.stdout:Epoch : 1 2023-12-16T20:33:58.920 INFO:teuthology.orchestra.run.smithi150.stdout:Version : 2.06 2023-12-16T20:33:58.920 INFO:teuthology.orchestra.run.smithi150.stdout:Release : 68.el9 2023-12-16T20:33:58.920 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: x86_64 2023-12-16T20:33:58.920 INFO:teuthology.orchestra.run.smithi150.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-16T20:33:58.921 INFO:teuthology.orchestra.run.smithi150.stdout:Group : Unspecified 2023-12-16T20:33:58.921 INFO:teuthology.orchestra.run.smithi150.stdout:Size : 8264254 2023-12-16T20:33:58.921 INFO:teuthology.orchestra.run.smithi150.stdout:License : GPLv3+ 2023-12-16T20:33:58.921 INFO:teuthology.orchestra.run.smithi150.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-16T20:33:58.921 INFO:teuthology.orchestra.run.smithi150.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-16T20:33:58.921 INFO:teuthology.orchestra.run.smithi150.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-16T20:33:58.921 INFO:teuthology.orchestra.run.smithi150.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-16T20:33:58.921 INFO:teuthology.orchestra.run.smithi150.stdout:Packager : builder@centos.org 2023-12-16T20:33:58.921 INFO:teuthology.orchestra.run.smithi150.stdout:Vendor : CentOS 2023-12-16T20:33:58.921 INFO:teuthology.orchestra.run.smithi150.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-16T20:33:58.921 INFO:teuthology.orchestra.run.smithi150.stdout:Summary : Support tools for GRUB. 2023-12-16T20:33:58.921 INFO:teuthology.orchestra.run.smithi150.stdout:Description : 2023-12-16T20:33:58.921 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:33:58.921 INFO:teuthology.orchestra.run.smithi150.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-16T20:33:58.922 INFO:teuthology.orchestra.run.smithi150.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-16T20:33:58.922 INFO:teuthology.orchestra.run.smithi150.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-16T20:33:58.922 INFO:teuthology.orchestra.run.smithi150.stdout:hardware devices. 2023-12-16T20:33:58.922 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:33:58.922 INFO:teuthology.orchestra.run.smithi150.stdout:This subpackage provides tools for support of all platforms. 2023-12-16T20:33:58.923 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-16T20:33:58.923 INFO:teuthology.task.kernel:Updating grub on smithi150 to boot 5.14.0-391.el9.x86_64 2023-12-16T20:33:58.923 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-16T20:33:59.062 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T20:33:59.210 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T20:33:59.211 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T20:33:59.211 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-16T20:33:59.211 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-16T20:33:59.211 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-16T20:33:59.211 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-12-16T20:33:59.211 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi052.front.sepia.ceph.com, path=None, version=distro) 2023-12-16T20:33:59.211 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T20:33:59.334 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T20:33:59.334 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T20:33:59.334 DEBUG:teuthology.orchestra.run.smithi052:> sudo rpm -qi grub2-tools 2023-12-16T20:33:59.381 INFO:teuthology.orchestra.run.smithi052.stdout:Name : grub2-tools 2023-12-16T20:33:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:Epoch : 1 2023-12-16T20:33:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:Version : 2.06 2023-12-16T20:33:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:Release : 68.el9 2023-12-16T20:33:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: x86_64 2023-12-16T20:33:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-16T20:33:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:Group : Unspecified 2023-12-16T20:33:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:Size : 8264254 2023-12-16T20:33:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:License : GPLv3+ 2023-12-16T20:33:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-16T20:33:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-16T20:33:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-16T20:33:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-16T20:33:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:Packager : builder@centos.org 2023-12-16T20:33:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:Vendor : CentOS 2023-12-16T20:33:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-16T20:33:59.383 INFO:teuthology.orchestra.run.smithi052.stdout:Summary : Support tools for GRUB. 2023-12-16T20:33:59.383 INFO:teuthology.orchestra.run.smithi052.stdout:Description : 2023-12-16T20:33:59.383 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:33:59.383 INFO:teuthology.orchestra.run.smithi052.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-16T20:33:59.383 INFO:teuthology.orchestra.run.smithi052.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-16T20:33:59.383 INFO:teuthology.orchestra.run.smithi052.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-16T20:33:59.383 INFO:teuthology.orchestra.run.smithi052.stdout:hardware devices. 2023-12-16T20:33:59.383 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:33:59.383 INFO:teuthology.orchestra.run.smithi052.stdout:This subpackage provides tools for support of all platforms. 2023-12-16T20:33:59.384 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-16T20:33:59.384 INFO:teuthology.task.kernel:Updating grub on smithi052 to boot 5.14.0-391.el9.x86_64 2023-12-16T20:33:59.384 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-16T20:33:59.580 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2023-12-16T20:33:59.991 INFO:teuthology.orchestra.run.smithi052.stderr:Generating grub configuration file ... 2023-12-16T20:34:01.051 INFO:teuthology.orchestra.run.smithi150.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-16T20:34:01.084 INFO:teuthology.orchestra.run.smithi150.stderr:done 2023-12-16T20:34:01.087 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2023-12-16T20:34:01.101 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.h3p1TZACRy 2023-12-16T20:34:01.102 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.h3p1TZACRy 2023-12-16T20:34:01.171 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.h3p1TZACRy 2023-12-16T20:34:01.320 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.h3p1TZACRy is 6KB 2023-12-16T20:34:01.368 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.h3p1TZACRy 2023-12-16T20:34:01.384 DEBUG:teuthology.orchestra.run.smithi150:> sudo /bin/ls /boot/loader/entries || true 2023-12-16T20:34:01.425 INFO:teuthology.orchestra.run.smithi052.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-16T20:34:01.452 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-16T20:34:01.452 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-16T20:34:01.452 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-16T20:34:01.453 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-16T20:34:01.454 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-16T20:34:01.460 INFO:teuthology.orchestra.run.smithi052.stderr:done 2023-12-16T20:34:01.463 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2023-12-16T20:34:01.519 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.OZsEUKawhl 2023-12-16T20:34:01.519 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OZsEUKawhl 2023-12-16T20:34:01.619 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2023-12-16T20:34:01.624 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.OZsEUKawhl 2023-12-16T20:34:01.730 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.OZsEUKawhl is 6KB 2023-12-16T20:34:01.779 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.OZsEUKawhl 2023-12-16T20:34:01.794 DEBUG:teuthology.orchestra.run.smithi052:> sudo /bin/ls /boot/loader/entries || true 2023-12-16T20:34:01.871 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-16T20:34:01.871 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-16T20:34:01.871 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-16T20:34:01.871 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-16T20:34:01.872 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-16T20:34:02.015 DEBUG:teuthology.orchestra.run.smithi052:> sudo shutdown -r now 2023-12-16T20:34:31.649 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-12-16T20:34:31.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T20:34:31.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T20:34:32.018 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-12-16T20:34:32.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-12-16T20:34:32.019 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T20:35:06.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T20:35:15.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T20:35:15.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T20:35:32.037 DEBUG:teuthology.orchestra.remote:timed out 2023-12-16T20:35:33.521 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T20:35:41.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-12-16T20:35:41.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T20:35:41.479 DEBUG:teuthology.orchestra.run.smithi052:> true 2023-12-16T20:35:41.869 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-12-16T20:35:41.869 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-391.el9.x86_64"... 2023-12-16T20:35:41.869 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2023-12-16T20:35:41.884 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-391.el9.x86_64 2023-12-16T20:35:41.884 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-16T20:35:41.884 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-16T20:35:41.884 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-16T20:35:42.885 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-12-16T20:35:42.886 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-16T20:35:42.946 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2023-12-16T20:35:42.984 DEBUG:teuthology.parallel:result is None 2023-12-16T20:35:45.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T20:35:45.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T20:35:45.957 DEBUG:teuthology.orchestra.run.smithi150:> true 2023-12-16T20:35:46.347 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T20:35:46.347 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-391.el9.x86_64"... 2023-12-16T20:35:46.348 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2023-12-16T20:35:46.362 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-391.el9.x86_64 2023-12-16T20:35:46.362 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-16T20:35:46.362 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-16T20:35:46.362 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-16T20:35:47.363 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-12-16T20:35:47.363 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-16T20:35:47.488 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2023-12-16T20:35:47.519 DEBUG:teuthology.parallel:result is None 2023-12-16T20:35:47.520 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-16T20:35:47.527 INFO:teuthology.task.internal:Creating test directory... 2023-12-16T20:35:47.527 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-16T20:35:47.530 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-16T20:35:47.556 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-16T20:35:47.592 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-16T20:35:47.602 INFO:teuthology.task.internal:Creating archive directory... 2023-12-16T20:35:47.603 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-16T20:35:47.605 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-16T20:35:47.646 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-16T20:35:47.652 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-16T20:35:47.653 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-16T20:35:47.683 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-16T20:35:47.718 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T20:35:47.730 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T20:35:47.731 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T20:35:47.757 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T20:35:47.759 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-16T20:35:47.766 INFO:teuthology.task.internal:Configuring sudo... 2023-12-16T20:35:47.767 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-16T20:35:47.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-16T20:35:47.820 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-16T20:35:47.829 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-16T20:35:47.829 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-16T20:35:47.832 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-16T20:35:47.855 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T20:35:47.948 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T20:35:48.007 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T20:35:48.087 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T20:35:48.136 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:35:48.136 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-16T20:35:48.209 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T20:35:48.250 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T20:35:48.302 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T20:35:48.383 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T20:35:48.433 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-16T20:35:48.433 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-16T20:35:48.503 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2023-12-16T20:35:48.505 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2023-12-16T20:35:48.602 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-16T20:35:48.656 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-16T20:35:48.823 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-16T20:35:48.833 INFO:teuthology.task.internal:Starting timer... 2023-12-16T20:35:48.833 INFO:teuthology.run_tasks:Running task pcp... 2023-12-16T20:35:48.852 INFO:teuthology.run_tasks:Running task selinux... 2023-12-16T20:35:48.861 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-16T20:35:48.861 DEBUG:teuthology.orchestra.run.smithi052:> sudo service auditd rotate 2023-12-16T20:35:48.924 INFO:teuthology.orchestra.run.smithi052.stdout:Rotating logs: 2023-12-16T20:35:48.925 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2023-12-16T20:35:48.980 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2023-12-16T20:35:48.981 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-16T20:35:48.982 DEBUG:teuthology.orchestra.run.smithi052:> /usr/sbin/getenforce 2023-12-16T20:35:49.007 INFO:teuthology.orchestra.run.smithi052.stdout:Permissive 2023-12-16T20:35:49.007 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2023-12-16T20:35:49.028 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2023-12-16T20:35:49.028 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi052.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi150.front.sepia.ceph.com': 'permissive'} 2023-12-16T20:35:49.029 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-16T20:35:49.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T20:35:49.080 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-16T20:35:49.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T20:35:49.108 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-16T20:35:49.108 DEBUG:teuthology.orchestra.run.smithi052:> sudo /usr/sbin/setenforce permissive 2023-12-16T20:35:49.158 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2023-12-16T20:35:49.197 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-16T20:35:49.208 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-16T20:35:49.214 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-16T20:35:49.353 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-16T20:35:49.370 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-16T20:35:49.371 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi052.front.sepia.ceph.com,smithi150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-16T20:41:31.927 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi052.front.sepia.ceph.com'), Remote(name='ubuntu@smithi150.front.sepia.ceph.com')] 2023-12-16T20:41:31.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-12-16T20:41:31.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T20:41:32.031 DEBUG:teuthology.orchestra.run.smithi052:> true 2023-12-16T20:41:32.101 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-12-16T20:41:32.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T20:41:32.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T20:41:32.186 DEBUG:teuthology.orchestra.run.smithi150:> true 2023-12-16T20:41:32.252 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T20:41:32.252 INFO:teuthology.run_tasks:Running task clock... 2023-12-16T20:41:32.264 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-16T20:41:32.264 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-16T20:41:32.265 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T20:41:32.267 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-16T20:41:32.267 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T20:41:32.301 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-16T20:41:32.322 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-16T20:41:32.323 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-16T20:41:32.339 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-16T20:41:32.359 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: ntpd: command not found 2023-12-16T20:41:32.368 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2023-12-16T20:41:32.376 INFO:teuthology.orchestra.run.smithi052.stdout:506 Cannot talk to daemon 2023-12-16T20:41:32.382 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2023-12-16T20:41:32.396 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-16T20:41:32.398 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-16T20:41:32.415 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-16T20:41:32.415 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-16T20:41:32.458 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2023-12-16T20:41:32.461 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-16T20:41:32.461 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2023-12-16T20:41:32.461 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T20:41:32.461 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T20:41:32.461 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T20:41:32.461 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T20:41:32.462 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2023-12-16T20:41:32.465 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-16T20:41:32.465 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2023-12-16T20:41:32.466 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T20:41:32.466 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T20:41:32.466 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T20:41:32.466 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T20:41:32.467 INFO:teuthology.run_tasks:Running task install... 2023-12-16T20:41:32.475 DEBUG:teuthology.task.install:project ceph 2023-12-16T20:41:32.475 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'}} 2023-12-16T20:41:32.475 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'} 2023-12-16T20:41:32.475 INFO:teuthology.task.install:Using flavor: default 2023-12-16T20:41:32.481 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-16T20:41:32.481 INFO:teuthology.task.install:extra packages: [] 2023-12-16T20:41:32.482 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'tag': None, 'wait_for_package': False} 2023-12-16T20:41:32.482 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T20:41:32.482 INFO:teuthology.packaging:ref: None 2023-12-16T20:41:32.482 INFO:teuthology.packaging:tag: None 2023-12-16T20:41:32.482 INFO:teuthology.packaging:branch: quincy 2023-12-16T20:41:32.482 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T20:41:32.483 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-16T20:41:32.485 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'tag': None, 'wait_for_package': False} 2023-12-16T20:41:32.485 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T20:41:32.485 INFO:teuthology.packaging:ref: None 2023-12-16T20:41:32.485 INFO:teuthology.packaging:tag: None 2023-12-16T20:41:32.485 INFO:teuthology.packaging:branch: quincy 2023-12-16T20:41:32.485 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T20:41:32.485 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-16T20:41:32.698 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/ 2023-12-16T20:41:32.699 INFO:teuthology.task.install.rpm:Package version is 17.2.7-125.g1f2f5510 2023-12-16T20:41:32.795 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/ 2023-12-16T20:41:32.795 INFO:teuthology.task.install.rpm:Package version is 17.2.7-125.g1f2f5510 2023-12-16T20:41:32.873 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-16T20:41:32.873 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-16T20:41:32.873 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-16T20:41:32.903 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-16T20:41:32.904 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T20:41:32.904 INFO:teuthology.packaging:ref: None 2023-12-16T20:41:32.904 INFO:teuthology.packaging:tag: None 2023-12-16T20:41:32.904 INFO:teuthology.packaging:branch: quincy 2023-12-16T20:41:32.904 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T20:41:32.904 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-16T20:41:32.962 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-16T20:41:32.962 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:41:32.962 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-16T20:41:32.973 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-16T20:41:32.995 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-16T20:41:32.995 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T20:41:32.995 INFO:teuthology.packaging:ref: None 2023-12-16T20:41:32.995 INFO:teuthology.packaging:tag: None 2023-12-16T20:41:32.995 INFO:teuthology.packaging:branch: quincy 2023-12-16T20:41:32.995 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T20:41:32.995 DEBUG:teuthology.orchestra.run.smithi052:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-16T20:41:33.055 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-16T20:41:33.068 DEBUG:teuthology.orchestra.run.smithi052:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-16T20:41:33.121 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2023-12-16T20:41:33.123 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2023-12-16T20:41:33.155 DEBUG:teuthology.orchestra.run.smithi052:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-16T20:41:33.185 INFO:teuthology.orchestra.run.smithi052.stdout:check_obsoletes = 1 2023-12-16T20:41:33.187 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum clean all 2023-12-16T20:41:33.375 INFO:teuthology.orchestra.run.smithi150.stdout:73 files removed 2023-12-16T20:41:33.402 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw 2023-12-16T20:41:33.453 INFO:teuthology.orchestra.run.smithi052.stdout:73 files removed 2023-12-16T20:41:33.479 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw 2023-12-16T20:41:34.126 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 191 kB/s | 77 kB 00:00 2023-12-16T20:41:34.214 INFO:teuthology.orchestra.run.smithi052.stdout:ceph packages for x86_64 194 kB/s | 77 kB 00:00 2023-12-16T20:41:34.532 INFO:teuthology.orchestra.run.smithi052.stdout:ceph noarch packages 34 kB/s | 9.1 kB 00:00 2023-12-16T20:41:34.551 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 24 kB/s | 9.1 kB 00:00 2023-12-16T20:41:34.902 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 5.7 kB/s | 1.8 kB 00:00 2023-12-16T20:41:34.966 INFO:teuthology.orchestra.run.smithi052.stdout:ceph source packages 4.5 kB/s | 1.8 kB 00:00 2023-12-16T20:41:35.404 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-12-16T20:41:35.445 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - AppStream 15 MB/s | 6.4 MB 00:00 2023-12-16T20:41:36.730 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 7.8 MB/s | 2.0 MB 00:00 2023-12-16T20:41:36.927 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2023-12-16T20:41:38.537 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 7.9 MB 00:01 2023-12-16T20:41:39.221 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 7.9 MB 00:01 2023-12-16T20:41:41.277 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 19 MB/s | 18 MB 00:00 2023-12-16T20:41:41.407 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 10 MB/s | 18 MB 00:01 2023-12-16T20:41:45.439 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 9.0 MB/s | 5.8 MB 00:00 2023-12-16T20:41:45.861 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 5.8 MB 00:01 2023-12-16T20:41:47.031 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2023-12-16T20:41:47.695 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 15 kB 00:00 2023-12-16T20:41:47.961 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 24 MB/s | 20 MB 00:00 2023-12-16T20:41:52.709 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2023-12-16T20:41:55.309 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:41:55.322 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:41:55.322 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-12-16T20:41:55.322 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:41:55.322 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:41:55.322 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 14 M 2023-12-16T20:41:55.322 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2023-12-16T20:41:55.322 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 3.1 M 2023-12-16T20:41:55.322 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 2.9 M 2023-12-16T20:41:55.322 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-12-16T20:41:55.322 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 5.0 M 2023-12-16T20:41:55.322 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 23 M 2023-12-16T20:41:55.323 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 24 k 2023-12-16T20:41:55.323 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-16T20:41:55.323 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 636 k 2023-12-16T20:41:55.323 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-16T20:41:55.323 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-16T20:41:55.323 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-16T20:41:55.323 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 458 k 2023-12-16T20:41:55.323 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-16T20:41:55.323 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 4.9 M 2023-12-16T20:41:55.323 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-16T20:41:55.323 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-16T20:41:55.323 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 45 k 2023-12-16T20:41:55.323 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 116 k 2023-12-16T20:41:55.323 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 156 k 2023-12-16T20:41:55.323 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-16T20:41:55.324 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 320 k 2023-12-16T20:41:55.324 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 292 k 2023-12-16T20:41:55.324 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 99 k 2023-12-16T20:41:55.324 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-16T20:41:55.324 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:41:55.324 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:41:55.324 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:41:55.324 INFO:teuthology.orchestra.run.smithi150.stdout:Install 22 Packages 2023-12-16T20:41:55.324 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 2 Packages 2023-12-16T20:41:55.324 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:41:55.325 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 58 M 2023-12-16T20:41:55.325 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:41:56.267 INFO:teuthology.orchestra.run.smithi150.stdout:(1/24): ceph-base-17.2.7-125.g1f2f5510.el9.x86_ 6.5 MB/s | 5.0 MB 00:00 2023-12-16T20:41:56.336 INFO:teuthology.orchestra.run.smithi150.stdout:(2/24): ceph-selinux-17.2.7-125.g1f2f5510.el9.x 361 kB/s | 24 kB 00:00 2023-12-16T20:41:56.443 INFO:teuthology.orchestra.run.smithi150.stdout:(3/24): libcephfs2-17.2.7-125.g1f2f5510.el9.x86 5.7 MB/s | 636 kB 00:00 2023-12-16T20:41:56.552 INFO:teuthology.orchestra.run.smithi150.stdout:(4/24): libradosstriper1-17.2.7-125.g1f2f5510.e 4.1 MB/s | 458 kB 00:00 2023-12-16T20:41:57.248 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 2.1 MB/s | 20 MB 00:09 2023-12-16T20:41:57.260 INFO:teuthology.orchestra.run.smithi150.stdout:(5/24): ceph-radosgw-17.2.7-125.g1f2f5510.el9.x 7.9 MB/s | 14 MB 00:01 2023-12-16T20:41:57.335 INFO:teuthology.orchestra.run.smithi150.stdout:(6/24): librgw2-17.2.7-125.g1f2f5510.el9.x86_64 6.3 MB/s | 4.9 MB 00:00 2023-12-16T20:41:57.352 INFO:teuthology.orchestra.run.smithi150.stdout:(7/24): python3-ceph-argparse-17.2.7-125.g1f2f5 491 kB/s | 45 kB 00:00 2023-12-16T20:41:57.395 INFO:teuthology.orchestra.run.smithi150.stdout:(8/24): python3-ceph-common-17.2.7-125.g1f2f551 1.9 MB/s | 116 kB 00:00 2023-12-16T20:41:57.420 INFO:teuthology.orchestra.run.smithi150.stdout:(9/24): python3-cephfs-17.2.7-125.g1f2f5510.el9 2.3 MB/s | 156 kB 00:00 2023-12-16T20:41:57.628 INFO:teuthology.orchestra.run.smithi150.stdout:(10/24): ceph-common-17.2.7-125.g1f2f5510.el9.x 11 MB/s | 23 MB 00:02 2023-12-16T20:41:57.653 INFO:teuthology.orchestra.run.smithi150.stdout:(11/24): python3-rados-17.2.7-125.g1f2f5510.el9 1.2 MB/s | 320 kB 00:00 2023-12-16T20:41:57.679 INFO:teuthology.orchestra.run.smithi150.stdout:(12/24): python3-rgw-17.2.7-125.g1f2f5510.el9.x 1.9 MB/s | 99 kB 00:00 2023-12-16T20:41:57.704 INFO:teuthology.orchestra.run.smithi150.stdout:(13/24): python3-rbd-17.2.7-125.g1f2f5510.el9.x 1.0 MB/s | 292 kB 00:00 2023-12-16T20:41:57.829 INFO:teuthology.orchestra.run.smithi150.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 294 kB/s | 44 kB 00:00 2023-12-16T20:41:57.854 INFO:teuthology.orchestra.run.smithi150.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 793 kB/s | 159 kB 00:00 2023-12-16T20:41:57.913 INFO:teuthology.orchestra.run.smithi150.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2023-12-16T20:41:57.945 INFO:teuthology.orchestra.run.smithi150.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 455 kB/s | 41 kB 00:00 2023-12-16T20:41:57.971 INFO:teuthology.orchestra.run.smithi150.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 661 kB 00:00 2023-12-16T20:41:58.563 INFO:teuthology.orchestra.run.smithi150.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 79 kB/s | 49 kB 00:00 2023-12-16T20:41:58.655 INFO:teuthology.orchestra.run.smithi150.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 98 kB/s | 67 kB 00:00 2023-12-16T20:41:58.763 INFO:teuthology.orchestra.run.smithi150.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 363 kB/s | 309 kB 00:00 2023-12-16T20:41:58.989 INFO:teuthology.orchestra.run.smithi150.stdout:(22/24): librados2-17.2.7-125.g1f2f5510.el9.x86 9.2 MB/s | 3.1 MB 00:00 2023-12-16T20:41:59.064 INFO:teuthology.orchestra.run.smithi150.stdout:(23/24): librbd1-17.2.7-125.g1f2f5510.el9.x86_6 9.8 MB/s | 2.9 MB 00:00 2023-12-16T20:41:59.347 INFO:teuthology.orchestra.run.smithi150.stdout:(24/24): thrift-0.15.0-2.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2023-12-16T20:41:59.352 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:41:59.352 INFO:teuthology.orchestra.run.smithi150.stdout:Total 14 MB/s | 58 MB 00:04 2023-12-16T20:41:59.353 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:41:59.386 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:41:59.387 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:41:59.858 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:41:59.858 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:42:01.075 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:42:01.434 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2023-12-16T20:42:01.508 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 2/26 2023-12-16T20:42:01.600 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 2/26 2023-12-16T20:42:01.687 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 3/26 2023-12-16T20:42:01.710 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 4/26 2023-12-16T20:42:01.798 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 4/26 2023-12-16T20:42:01.842 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2023-12-16T20:42:01.873 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2023-12-16T20:42:02.120 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2023-12-16T20:42:02.312 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2023-12-16T20:42:02.346 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 8/26 2023-12-16T20:42:02.391 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 8/26 2023-12-16T20:42:02.421 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2023-12-16T20:42:02.473 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:17.2.7-125.g1f2f5510.el9.x 10/26 2023-12-16T20:42:02.509 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 11/26 2023-12-16T20:42:02.571 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 12/26 2023-12-16T20:42:02.599 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 13/26 2023-12-16T20:42:02.648 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 13/26 2023-12-16T20:42:02.688 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2023-12-16T20:42:02.724 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2023-12-16T20:42:02.756 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2023-12-16T20:42:03.049 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2023-12-16T20:42:03.079 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 18/26 2023-12-16T20:42:03.153 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 18/26 2023-12-16T20:42:03.225 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 19/26 2023-12-16T20:42:03.249 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:17.2.7-125.g1f2f5510.el9.x86 20/26 2023-12-16T20:42:04.681 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:42:04.684 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:42:04.684 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-12-16T20:42:04.684 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:42:04.684 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:42:04.684 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 14 M 2023-12-16T20:42:04.684 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrading: 2023-12-16T20:42:04.684 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 3.1 M 2023-12-16T20:42:04.684 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 2.9 M 2023-12-16T20:42:04.685 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-12-16T20:42:04.685 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 5.0 M 2023-12-16T20:42:04.685 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 23 M 2023-12-16T20:42:04.685 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 24 k 2023-12-16T20:42:04.685 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-16T20:42:04.685 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 636 k 2023-12-16T20:42:04.685 INFO:teuthology.orchestra.run.smithi052.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-16T20:42:04.685 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-16T20:42:04.685 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-16T20:42:04.685 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 458 k 2023-12-16T20:42:04.685 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-16T20:42:04.685 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 4.9 M 2023-12-16T20:42:04.685 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-16T20:42:04.685 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-16T20:42:04.685 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 45 k 2023-12-16T20:42:04.686 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 116 k 2023-12-16T20:42:04.686 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 156 k 2023-12-16T20:42:04.686 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-16T20:42:04.686 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 320 k 2023-12-16T20:42:04.686 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 292 k 2023-12-16T20:42:04.686 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 99 k 2023-12-16T20:42:04.686 INFO:teuthology.orchestra.run.smithi052.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-16T20:42:04.686 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:04.686 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:42:04.686 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:42:04.686 INFO:teuthology.orchestra.run.smithi052.stdout:Install 22 Packages 2023-12-16T20:42:04.686 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrade 2 Packages 2023-12-16T20:42:04.686 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:04.687 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 58 M 2023-12-16T20:42:04.687 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:42:05.630 INFO:teuthology.orchestra.run.smithi052.stdout:(1/24): ceph-base-17.2.7-125.g1f2f5510.el9.x86_ 8.7 MB/s | 5.0 MB 00:00 2023-12-16T20:42:05.688 INFO:teuthology.orchestra.run.smithi052.stdout:(2/24): ceph-selinux-17.2.7-125.g1f2f5510.el9.x 414 kB/s | 24 kB 00:00 2023-12-16T20:42:05.698 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 21/26 2023-12-16T20:42:05.728 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 21/26 2023-12-16T20:42:05.772 INFO:teuthology.orchestra.run.smithi052.stdout:(3/24): libcephfs2-17.2.7-125.g1f2f5510.el9.x86 7.5 MB/s | 636 kB 00:00 2023-12-16T20:42:05.847 INFO:teuthology.orchestra.run.smithi052.stdout:(4/24): libradosstriper1-17.2.7-125.g1f2f5510.e 6.0 MB/s | 458 kB 00:00 2023-12-16T20:42:06.264 INFO:teuthology.orchestra.run.smithi052.stdout:(5/24): librgw2-17.2.7-125.g1f2f5510.el9.x86_64 12 MB/s | 4.9 MB 00:00 2023-12-16T20:42:06.276 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 21/26 2023-12-16T20:42:06.307 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 22/26 2023-12-16T20:42:06.331 INFO:teuthology.orchestra.run.smithi052.stdout:(6/24): python3-ceph-argparse-17.2.7-125.g1f2f5 678 kB/s | 45 kB 00:00 2023-12-16T20:42:06.393 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 22/26 2023-12-16T20:42:06.393 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-16T20:42:06.394 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-16T20:42:06.394 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:06.415 INFO:teuthology.orchestra.run.smithi052.stdout:(7/24): python3-ceph-common-17.2.7-125.g1f2f551 1.4 MB/s | 116 kB 00:00 2023-12-16T20:42:06.425 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 23/26 2023-12-16T20:42:06.498 INFO:teuthology.orchestra.run.smithi052.stdout:(8/24): python3-cephfs-17.2.7-125.g1f2f5510.el9 1.8 MB/s | 156 kB 00:00 2023-12-16T20:42:06.599 INFO:teuthology.orchestra.run.smithi052.stdout:(9/24): python3-rados-17.2.7-125.g1f2f5510.el9. 3.1 MB/s | 320 kB 00:00 2023-12-16T20:42:06.707 INFO:teuthology.orchestra.run.smithi052.stdout:(10/24): python3-rbd-17.2.7-125.g1f2f5510.el9.x 2.6 MB/s | 292 kB 00:00 2023-12-16T20:42:06.949 INFO:teuthology.orchestra.run.smithi052.stdout:(11/24): ceph-common-17.2.7-125.g1f2f5510.el9.x 12 MB/s | 23 MB 00:01 2023-12-16T20:42:06.977 INFO:teuthology.orchestra.run.smithi052.stdout:(12/24): python3-rgw-17.2.7-125.g1f2f5510.el9.x 372 kB/s | 99 kB 00:00 2023-12-16T20:42:07.083 INFO:teuthology.orchestra.run.smithi052.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 545 kB/s | 44 kB 00:00 2023-12-16T20:42:07.108 INFO:teuthology.orchestra.run.smithi052.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2023-12-16T20:42:07.167 INFO:teuthology.orchestra.run.smithi052.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2023-12-16T20:42:07.192 INFO:teuthology.orchestra.run.smithi052.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2023-12-16T20:42:07.209 INFO:teuthology.orchestra.run.smithi052.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 992 kB/s | 41 kB 00:00 2023-12-16T20:42:07.385 INFO:teuthology.orchestra.run.smithi052.stdout:(18/24): ceph-radosgw-17.2.7-125.g1f2f5510.el9. 6.0 MB/s | 14 MB 00:02 2023-12-16T20:42:07.693 INFO:teuthology.orchestra.run.smithi052.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 100 kB/s | 49 kB 00:00 2023-12-16T20:42:07.752 INFO:teuthology.orchestra.run.smithi052.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 215 kB/s | 67 kB 00:00 2023-12-16T20:42:07.777 INFO:teuthology.orchestra.run.smithi052.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 528 kB/s | 309 kB 00:00 2023-12-16T20:42:07.986 INFO:teuthology.orchestra.run.smithi052.stdout:(22/24): librados2-17.2.7-125.g1f2f5510.el9.x86 13 MB/s | 3.1 MB 00:00 2023-12-16T20:42:08.028 INFO:teuthology.orchestra.run.smithi052.stdout:(23/24): thrift-0.15.0-2.el9.x86_64.rpm 4.7 MB/s | 1.6 MB 00:00 2023-12-16T20:42:08.086 INFO:teuthology.orchestra.run.smithi052.stdout:(24/24): librbd1-17.2.7-125.g1f2f5510.el9.x86_6 9.5 MB/s | 2.9 MB 00:00 2023-12-16T20:42:08.094 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:42:08.094 INFO:teuthology.orchestra.run.smithi052.stdout:Total 17 MB/s | 58 MB 00:03 2023-12-16T20:42:08.094 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:42:08.131 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:42:08.131 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:42:08.608 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:42:08.608 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:42:09.879 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:42:10.239 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2023-12-16T20:42:10.297 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 2/26 2023-12-16T20:42:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 2/26 2023-12-16T20:42:10.455 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 3/26 2023-12-16T20:42:10.482 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 4/26 2023-12-16T20:42:10.553 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 4/26 2023-12-16T20:42:10.608 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2023-12-16T20:42:10.639 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2023-12-16T20:42:11.110 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2023-12-16T20:42:11.208 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 8/26 2023-12-16T20:42:11.316 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 8/26 2023-12-16T20:42:11.345 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2023-12-16T20:42:11.394 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-argparse-2:17.2.7-125.g1f2f5510.el9.x 10/26 2023-12-16T20:42:11.435 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 11/26 2023-12-16T20:42:11.499 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 12/26 2023-12-16T20:42:11.523 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 13/26 2023-12-16T20:42:11.572 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 13/26 2023-12-16T20:42:11.612 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2023-12-16T20:42:11.656 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2023-12-16T20:42:11.697 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2023-12-16T20:42:12.020 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2023-12-16T20:42:12.060 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 18/26 2023-12-16T20:42:12.151 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 18/26 2023-12-16T20:42:12.226 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 19/26 2023-12-16T20:42:12.255 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-common-2:17.2.7-125.g1f2f5510.el9.x86 20/26 2023-12-16T20:42:14.784 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 21/26 2023-12-16T20:42:14.826 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 21/26 2023-12-16T20:42:15.398 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 21/26 2023-12-16T20:42:15.440 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 22/26 2023-12-16T20:42:15.530 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 22/26 2023-12-16T20:42:15.530 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-16T20:42:15.530 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-16T20:42:15.530 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:15.574 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 23/26 2023-12-16T20:42:15.883 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 23/26 2023-12-16T20:42:15.883 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2023-12-16T20:42:15.883 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2023-12-16T20:42:15.883 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2023-12-16T20:42:15.883 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2023-12-16T20:42:15.883 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2023-12-16T20:42:15.884 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /root 2023-12-16T20:42:15.884 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2023-12-16T20:42:15.884 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:17.186 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 24/26 2023-12-16T20:42:17.264 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 24/26 2023-12-16T20:42:17.264 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T20:42:17.264 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-16T20:42:17.264 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T20:42:17.264 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T20:42:17.264 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:17.266 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-16T20:42:17.380 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-16T20:42:17.380 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-16T20:42:18.598 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-16T20:42:18.598 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 1/26 2023-12-16T20:42:18.598 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 2/26 2023-12-16T20:42:18.598 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 3/26 2023-12-16T20:42:18.598 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 4/26 2023-12-16T20:42:18.598 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 5/26 2023-12-16T20:42:18.598 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 6/26 2023-12-16T20:42:18.598 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 7/26 2023-12-16T20:42:18.598 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:17.2.7-125.g1f2f5510.el9.x 8/26 2023-12-16T20:42:18.598 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:17.2.7-125.g1f2f5510.el9.x86 9/26 2023-12-16T20:42:18.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 10/26 2023-12-16T20:42:18.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 11/26 2023-12-16T20:42:18.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 12/26 2023-12-16T20:42:18.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 13/26 2023-12-16T20:42:18.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2023-12-16T20:42:18.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2023-12-16T20:42:18.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2023-12-16T20:42:18.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2023-12-16T20:42:18.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2023-12-16T20:42:18.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2023-12-16T20:42:18.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2023-12-16T20:42:18.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2023-12-16T20:42:18.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2023-12-16T20:42:18.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 23/26 2023-12-16T20:42:18.599 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2023-12-16T20:42:18.600 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 25/26 2023-12-16T20:42:19.634 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2023-12-16T20:42:19.634 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:19.635 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2023-12-16T20:42:19.635 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:19.635 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:19.635 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:42:19.635 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:19.635 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:19.635 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:19.635 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:19.635 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-16T20:42:19.635 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:19.635 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-16T20:42:19.635 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-16T20:42:19.635 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-16T20:42:19.635 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:19.636 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-16T20:42:19.636 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:19.636 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-16T20:42:19.636 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-16T20:42:19.636 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:19.636 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:19.636 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:19.636 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-16T20:42:19.636 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:19.636 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:19.636 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:19.636 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-16T20:42:19.636 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:19.636 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:42:20.008 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-test 2023-12-16T20:42:20.563 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:28 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:42:20.718 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:42:20.718 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:42:20.718 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-12-16T20:42:20.718 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:42:20.719 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:42:20.719 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 48 M 2023-12-16T20:42:20.719 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-12-16T20:42:20.719 INFO:teuthology.orchestra.run.smithi150.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-16T20:42:20.719 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 159 k 2023-12-16T20:42:20.719 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-16T20:42:20.719 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-16T20:42:20.719 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-16T20:42:20.719 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:20.719 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:42:20.719 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:42:20.719 INFO:teuthology.orchestra.run.smithi150.stdout:Install 6 Packages 2023-12-16T20:42:20.719 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:20.720 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 49 M 2023-12-16T20:42:20.720 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 195 M 2023-12-16T20:42:20.720 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:42:20.884 INFO:teuthology.orchestra.run.smithi150.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 186 kB 00:00 2023-12-16T20:42:20.925 INFO:teuthology.orchestra.run.smithi150.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-12-16T20:42:20.968 INFO:teuthology.orchestra.run.smithi150.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-12-16T20:42:21.001 INFO:teuthology.orchestra.run.smithi150.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-12-16T20:42:21.035 INFO:teuthology.orchestra.run.smithi150.stdout:(5/6): libcephsqlite-17.2.7-125.g1f2f5510.el9.x 506 kB/s | 159 kB 00:00 2023-12-16T20:42:23.110 INFO:teuthology.orchestra.run.smithi150.stdout:(6/6): ceph-test-17.2.7-125.g1f2f5510.el9.x86_6 20 MB/s | 48 MB 00:02 2023-12-16T20:42:23.113 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:42:23.113 INFO:teuthology.orchestra.run.smithi150.stdout:Total 20 MB/s | 49 MB 00:02 2023-12-16T20:42:23.114 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:42:23.133 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:42:23.133 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:42:23.536 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:42:23.536 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:42:24.123 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:42:24.211 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-16T20:42:24.248 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-16T20:42:24.284 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-16T20:42:24.326 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-16T20:42:24.355 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 5/6 2023-12-16T20:42:25.442 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 23/26 2023-12-16T20:42:25.443 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /sys 2023-12-16T20:42:25.443 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /proc 2023-12-16T20:42:25.443 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /mnt 2023-12-16T20:42:25.443 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /var/tmp 2023-12-16T20:42:25.443 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /home 2023-12-16T20:42:25.443 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /root 2023-12-16T20:42:25.443 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /tmp 2023-12-16T20:42:25.443 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:26.782 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 24/26 2023-12-16T20:42:26.877 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 24/26 2023-12-16T20:42:26.878 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T20:42:26.878 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-16T20:42:26.878 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T20:42:26.878 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T20:42:26.878 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:26.879 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-16T20:42:26.960 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-16T20:42:26.960 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-16T20:42:28.198 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-16T20:42:28.199 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 1/26 2023-12-16T20:42:28.199 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 2/26 2023-12-16T20:42:28.199 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 3/26 2023-12-16T20:42:28.199 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 4/26 2023-12-16T20:42:28.199 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 5/26 2023-12-16T20:42:28.199 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 6/26 2023-12-16T20:42:28.199 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 7/26 2023-12-16T20:42:28.199 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-argparse-2:17.2.7-125.g1f2f5510.el9.x 8/26 2023-12-16T20:42:28.199 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-common-2:17.2.7-125.g1f2f5510.el9.x86 9/26 2023-12-16T20:42:28.199 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 10/26 2023-12-16T20:42:28.199 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 11/26 2023-12-16T20:42:28.199 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 12/26 2023-12-16T20:42:28.199 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 13/26 2023-12-16T20:42:28.199 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2023-12-16T20:42:28.200 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2023-12-16T20:42:28.200 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2023-12-16T20:42:28.200 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2023-12-16T20:42:28.200 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2023-12-16T20:42:28.200 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2023-12-16T20:42:28.200 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2023-12-16T20:42:28.200 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2023-12-16T20:42:28.200 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2023-12-16T20:42:28.200 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 23/26 2023-12-16T20:42:28.200 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2023-12-16T20:42:28.200 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 25/26 2023-12-16T20:42:28.744 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 5/6 2023-12-16T20:42:28.820 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 6/6 2023-12-16T20:42:29.169 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2023-12-16T20:42:29.169 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:29.169 INFO:teuthology.orchestra.run.smithi052.stdout:Upgraded: 2023-12-16T20:42:29.169 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:29.169 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:29.169 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:42:29.169 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:29.169 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:29.169 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:29.169 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:29.169 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-16T20:42:29.170 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:29.170 INFO:teuthology.orchestra.run.smithi052.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-16T20:42:29.170 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-16T20:42:29.170 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-16T20:42:29.170 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:29.170 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-16T20:42:29.170 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:29.170 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-16T20:42:29.170 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-16T20:42:29.170 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:29.170 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:29.170 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:29.170 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-16T20:42:29.170 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:29.170 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:29.171 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:29.171 INFO:teuthology.orchestra.run.smithi052.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-16T20:42:29.171 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:29.171 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:42:29.519 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-test 2023-12-16T20:42:30.102 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:28 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:42:30.255 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:42:30.256 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:42:30.256 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-12-16T20:42:30.256 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:42:30.256 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:42:30.256 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 48 M 2023-12-16T20:42:30.256 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-12-16T20:42:30.256 INFO:teuthology.orchestra.run.smithi052.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-16T20:42:30.256 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 159 k 2023-12-16T20:42:30.256 INFO:teuthology.orchestra.run.smithi052.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-16T20:42:30.256 INFO:teuthology.orchestra.run.smithi052.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-16T20:42:30.256 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-16T20:42:30.256 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:30.256 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:42:30.257 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:42:30.257 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2023-12-16T20:42:30.257 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:30.257 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 49 M 2023-12-16T20:42:30.257 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 195 M 2023-12-16T20:42:30.257 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:42:30.429 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 186 kB 00:00 2023-12-16T20:42:30.471 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-12-16T20:42:30.504 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 8.8 MB/s | 300 kB 00:00 2023-12-16T20:42:30.530 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2023-12-16T20:42:30.555 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): libcephsqlite-17.2.7-125.g1f2f5510.el9.x 535 kB/s | 159 kB 00:00 2023-12-16T20:42:31.078 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 6/6 2023-12-16T20:42:31.078 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 1/6 2023-12-16T20:42:31.078 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 2/6 2023-12-16T20:42:31.078 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-16T20:42:31.078 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-16T20:42:31.078 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-16T20:42:31.449 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-16T20:42:31.449 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:31.449 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:42:31.449 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:31.449 INFO:teuthology.orchestra.run.smithi150.stdout: jq-1.6-16.el9.x86_64 2023-12-16T20:42:31.449 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:31.449 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-16T20:42:31.450 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-16T20:42:31.450 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-16T20:42:31.450 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:31.450 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:42:31.735 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph 2023-12-16T20:42:32.291 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:40 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:42:32.444 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:42:32.447 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2023-12-16T20:42:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-12-16T20:42:32.447 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2023-12-16T20:42:32.447 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:42:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 6.5 k 2023-12-16T20:42:32.447 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-12-16T20:42:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 2.1 M 2023-12-16T20:42:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 1.4 M 2023-12-16T20:42:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 238 k 2023-12-16T20:42:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 3.7 M 2023-12-16T20:42:32.447 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 15 M 2023-12-16T20:42:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-16T20:42:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-16T20:42:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-16T20:42:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-16T20:42:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-16T20:42:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-16T20:42:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-16T20:42:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-16T20:42:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-16T20:42:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-16T20:42:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-16T20:42:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-16T20:42:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-16T20:42:32.448 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-16T20:42:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-16T20:42:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-16T20:42:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-16T20:42:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-16T20:42:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-16T20:42:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-16T20:42:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-16T20:42:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-16T20:42:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-16T20:42:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-16T20:42:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-16T20:42:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-16T20:42:32.449 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-16T20:42:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:32.450 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:42:32.450 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2023-12-16T20:42:32.450 INFO:teuthology.orchestra.run.smithi150.stdout:Install 33 Packages 2023-12-16T20:42:32.450 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:32.450 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 27 M 2023-12-16T20:42:32.450 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 99 M 2023-12-16T20:42:32.450 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:42:32.613 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): ceph-test-17.2.7-125.g1f2f5510.el9.x86_6 20 MB/s | 48 MB 00:02 2023-12-16T20:42:32.617 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:42:32.617 INFO:teuthology.orchestra.run.smithi052.stdout:Total 21 MB/s | 49 MB 00:02 2023-12-16T20:42:32.617 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:42:32.639 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:42:32.639 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:42:32.662 INFO:teuthology.orchestra.run.smithi150.stdout:(1/33): ceph-17.2.7-125.g1f2f5510.el9.x86_64.rp 41 kB/s | 6.5 kB 00:00 2023-12-16T20:42:32.887 INFO:teuthology.orchestra.run.smithi150.stdout:(2/33): ceph-mds-17.2.7-125.g1f2f5510.el9.x86_6 5.4 MB/s | 2.1 MB 00:00 2023-12-16T20:42:32.921 INFO:teuthology.orchestra.run.smithi150.stdout:(3/33): ceph-mgr-17.2.7-125.g1f2f5510.el9.x86_6 3.4 MB/s | 1.4 MB 00:00 2023-12-16T20:42:33.005 INFO:teuthology.orchestra.run.smithi150.stdout:(4/33): ceph-mgr-modules-core-17.2.7-125.g1f2f5 2.8 MB/s | 238 kB 00:00 2023-12-16T20:42:33.056 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:42:33.056 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:42:33.096 INFO:teuthology.orchestra.run.smithi150.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 144 kB 00:00 2023-12-16T20:42:33.130 INFO:teuthology.orchestra.run.smithi150.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 948 kB/s | 32 kB 00:00 2023-12-16T20:42:33.164 INFO:teuthology.orchestra.run.smithi150.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-12-16T20:42:33.197 INFO:teuthology.orchestra.run.smithi150.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-12-16T20:42:33.239 INFO:teuthology.orchestra.run.smithi150.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-12-16T20:42:33.331 INFO:teuthology.orchestra.run.smithi150.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2023-12-16T20:42:33.365 INFO:teuthology.orchestra.run.smithi150.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-12-16T20:42:33.423 INFO:teuthology.orchestra.run.smithi150.stdout:(12/33): ceph-mon-17.2.7-125.g1f2f5510.el9.x86_ 4.9 MB/s | 3.7 MB 00:00 2023-12-16T20:42:33.449 INFO:teuthology.orchestra.run.smithi150.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2023-12-16T20:42:33.474 INFO:teuthology.orchestra.run.smithi150.stdout:(14/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-16T20:42:33.499 INFO:teuthology.orchestra.run.smithi150.stdout:(15/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2023-12-16T20:42:33.660 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:42:33.683 INFO:teuthology.orchestra.run.smithi150.stdout:(16/33): ceph-osd-17.2.7-125.g1f2f5510.el9.x86_ 19 MB/s | 15 MB 00:00 2023-12-16T20:42:33.735 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-16T20:42:33.788 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-16T20:42:33.821 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-16T20:42:33.864 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-16T20:42:33.891 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 5/6 2023-12-16T20:42:34.067 INFO:teuthology.orchestra.run.smithi150.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 73 kB/s | 43 kB 00:00 2023-12-16T20:42:34.192 INFO:teuthology.orchestra.run.smithi150.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 85 kB/s | 11 kB 00:00 2023-12-16T20:42:34.309 INFO:teuthology.orchestra.run.smithi150.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2023-12-16T20:42:34.342 INFO:teuthology.orchestra.run.smithi150.stdout:(20/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 204 kB/s | 172 kB 00:00 2023-12-16T20:42:34.426 INFO:teuthology.orchestra.run.smithi150.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 198 kB/s | 23 kB 00:00 2023-12-16T20:42:34.459 INFO:teuthology.orchestra.run.smithi150.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 166 kB/s | 19 kB 00:00 2023-12-16T20:42:34.543 INFO:teuthology.orchestra.run.smithi150.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 168 kB/s | 20 kB 00:00 2023-12-16T20:42:34.577 INFO:teuthology.orchestra.run.smithi150.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 395 kB/s | 46 kB 00:00 2023-12-16T20:42:34.610 INFO:teuthology.orchestra.run.smithi150.stdout:(25/33): python3-cherrypy-18.6.1-2.el9.noarch.r 387 kB/s | 358 kB 00:00 2023-12-16T20:42:34.727 INFO:teuthology.orchestra.run.smithi150.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 141 kB/s | 16 kB 00:00 2023-12-16T20:42:34.836 INFO:teuthology.orchestra.run.smithi150.stdout:(27/33): python3-more-itertools-8.12.0-2.el9.no 269 kB/s | 79 kB 00:00 2023-12-16T20:42:34.861 INFO:teuthology.orchestra.run.smithi150.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 673 kB/s | 90 kB 00:00 2023-12-16T20:42:34.953 INFO:teuthology.orchestra.run.smithi150.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 306 kB/s | 36 kB 00:00 2023-12-16T20:42:34.978 INFO:teuthology.orchestra.run.smithi150.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 679 kB/s | 272 kB 00:00 2023-12-16T20:42:35.070 INFO:teuthology.orchestra.run.smithi150.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2023-12-16T20:42:35.097 INFO:teuthology.orchestra.run.smithi150.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2023-12-16T20:42:35.612 INFO:teuthology.orchestra.run.smithi150.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 648 kB/s | 427 kB 00:00 2023-12-16T20:42:35.637 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:42:35.637 INFO:teuthology.orchestra.run.smithi150.stdout:Total 8.4 MB/s | 27 MB 00:03 2023-12-16T20:42:35.637 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:42:35.655 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:42:35.655 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:42:35.998 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:42:35.999 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:42:37.223 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:42:37.307 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-12-16T20:42:37.339 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-12-16T20:42:37.372 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-12-16T20:42:37.464 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-12-16T20:42:37.542 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-12-16T20:42:37.582 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-12-16T20:42:37.654 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-12-16T20:42:37.709 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/33 2023-12-16T20:42:37.747 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2023-12-16T20:42:37.796 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2023-12-16T20:42:37.856 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2023-12-16T20:42:38.054 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2023-12-16T20:42:38.101 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2023-12-16T20:42:38.139 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2023-12-16T20:42:38.182 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2023-12-16T20:42:38.212 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2023-12-16T20:42:38.290 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2023-12-16T20:42:38.331 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-16T20:42:38.364 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2023-12-16T20:42:38.405 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2023-12-16T20:42:38.439 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2023-12-16T20:42:38.474 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-12-16T20:42:38.543 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 5/6 2023-12-16T20:42:38.563 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-12-16T20:42:38.605 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-12-16T20:42:38.618 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 6/6 2023-12-16T20:42:38.678 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-12-16T20:42:38.824 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-12-16T20:42:38.948 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-16T20:42:39.106 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-125.g1f2f5510.el9.n 28/33 2023-12-16T20:42:39.135 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 29/33 2023-12-16T20:42:39.159 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 29/33 2023-12-16T20:42:39.159 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T20:42:39.159 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-16T20:42:39.159 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T20:42:39.159 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T20:42:39.159 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:40.502 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 30/33 2023-12-16T20:42:40.531 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 30/33 2023-12-16T20:42:40.531 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T20:42:40.532 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-16T20:42:40.532 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T20:42:40.532 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T20:42:40.532 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:40.921 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 31/33 2023-12-16T20:42:40.944 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 31/33 2023-12-16T20:42:40.945 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T20:42:40.945 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-16T20:42:40.945 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T20:42:40.945 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T20:42:40.945 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:41.111 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 6/6 2023-12-16T20:42:41.111 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 1/6 2023-12-16T20:42:41.111 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 2/6 2023-12-16T20:42:41.111 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-16T20:42:41.111 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-16T20:42:41.111 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-16T20:42:41.155 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 32/33 2023-12-16T20:42:41.178 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 32/33 2023-12-16T20:42:41.178 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T20:42:41.178 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-16T20:42:41.178 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T20:42:41.178 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T20:42:41.178 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:41.245 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 33/33 2023-12-16T20:42:41.453 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-16T20:42:41.453 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:41.453 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:42:41.454 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:41.454 INFO:teuthology.orchestra.run.smithi052.stdout: jq-1.6-16.el9.x86_64 2023-12-16T20:42:41.454 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:41.454 INFO:teuthology.orchestra.run.smithi052.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-16T20:42:41.454 INFO:teuthology.orchestra.run.smithi052.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-16T20:42:41.454 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-16T20:42:41.454 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:41.454 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:42:41.725 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph 2023-12-16T20:42:42.309 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:40 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:42:42.468 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:42:42.470 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2023-12-16T20:42:42.470 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-12-16T20:42:42.470 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2023-12-16T20:42:42.470 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:42:42.470 INFO:teuthology.orchestra.run.smithi052.stdout: ceph x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 6.5 k 2023-12-16T20:42:42.470 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-12-16T20:42:42.470 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 2.1 M 2023-12-16T20:42:42.471 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 1.4 M 2023-12-16T20:42:42.471 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 238 k 2023-12-16T20:42:42.471 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 3.7 M 2023-12-16T20:42:42.471 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 15 M 2023-12-16T20:42:42.471 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-16T20:42:42.471 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-16T20:42:42.471 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-16T20:42:42.471 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-16T20:42:42.471 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-16T20:42:42.471 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-16T20:42:42.471 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-16T20:42:42.471 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-16T20:42:42.471 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-16T20:42:42.471 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-16T20:42:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-16T20:42:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-16T20:42:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-16T20:42:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-16T20:42:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-16T20:42:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-16T20:42:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-16T20:42:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-16T20:42:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-16T20:42:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-16T20:42:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-16T20:42:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-16T20:42:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-16T20:42:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-16T20:42:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-16T20:42:42.473 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-16T20:42:42.473 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-16T20:42:42.473 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:42.473 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:42:42.473 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2023-12-16T20:42:42.473 INFO:teuthology.orchestra.run.smithi052.stdout:Install 33 Packages 2023-12-16T20:42:42.473 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:42.474 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 27 M 2023-12-16T20:42:42.474 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 99 M 2023-12-16T20:42:42.474 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:42:42.820 INFO:teuthology.orchestra.run.smithi052.stdout:(1/33): ceph-17.2.7-125.g1f2f5510.el9.x86_64.rp 36 kB/s | 6.5 kB 00:00 2023-12-16T20:42:42.826 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 33/33 2023-12-16T20:42:42.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 1/33 2023-12-16T20:42:42.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 2/33 2023-12-16T20:42:42.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 3/33 2023-12-16T20:42:42.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 4/33 2023-12-16T20:42:42.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 5/33 2023-12-16T20:42:42.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-125.g1f2f5510.el9.n 6/33 2023-12-16T20:42:42.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-12-16T20:42:42.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-12-16T20:42:42.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2023-12-16T20:42:42.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2023-12-16T20:42:42.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2023-12-16T20:42:42.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2023-12-16T20:42:42.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2023-12-16T20:42:42.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2023-12-16T20:42:42.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2023-12-16T20:42:42.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 16/33 2023-12-16T20:42:42.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-12-16T20:42:42.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-16T20:42:42.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-12-16T20:42:42.827 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-12-16T20:42:42.828 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-12-16T20:42:42.828 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-12-16T20:42:42.828 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-12-16T20:42:42.828 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-12-16T20:42:42.828 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-12-16T20:42:42.829 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-12-16T20:42:42.829 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-16T20:42:42.829 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-12-16T20:42:42.829 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-12-16T20:42:42.829 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-12-16T20:42:42.829 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-12-16T20:42:42.829 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-12-16T20:42:43.004 INFO:teuthology.orchestra.run.smithi052.stdout:(2/33): ceph-mds-17.2.7-125.g1f2f5510.el9.x86_6 5.6 MB/s | 2.1 MB 00:00 2023-12-16T20:42:43.071 INFO:teuthology.orchestra.run.smithi052.stdout:(3/33): ceph-mgr-17.2.7-125.g1f2f5510.el9.x86_6 3.2 MB/s | 1.4 MB 00:00 2023-12-16T20:42:43.163 INFO:teuthology.orchestra.run.smithi052.stdout:(4/33): ceph-mgr-modules-core-17.2.7-125.g1f2f5 2.5 MB/s | 238 kB 00:00 2023-12-16T20:42:43.272 INFO:teuthology.orchestra.run.smithi052.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2023-12-16T20:42:43.305 INFO:teuthology.orchestra.run.smithi052.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 956 kB/s | 32 kB 00:00 2023-12-16T20:42:43.339 INFO:teuthology.orchestra.run.smithi052.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-12-16T20:42:43.372 INFO:teuthology.orchestra.run.smithi052.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-12-16T20:42:43.414 INFO:teuthology.orchestra.run.smithi052.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-12-16T20:42:43.490 INFO:teuthology.orchestra.run.smithi052.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2023-12-16T20:42:43.523 INFO:teuthology.orchestra.run.smithi052.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-12-16T20:42:43.557 INFO:teuthology.orchestra.run.smithi052.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-12-16T20:42:43.590 INFO:teuthology.orchestra.run.smithi052.stdout:(13/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-12-16T20:42:43.624 INFO:teuthology.orchestra.run.smithi052.stdout:(14/33): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-12-16T20:42:43.710 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-12-16T20:42:43.711 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:43.711 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:42:43.711 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:43.711 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:43.711 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:43.712 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:42:43.712 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:43.712 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:43.712 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-16T20:42:43.712 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-16T20:42:43.712 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-16T20:42:43.712 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-16T20:42:43.712 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-16T20:42:43.712 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-16T20:42:43.712 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-16T20:42:43.712 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-16T20:42:43.712 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-16T20:42:43.713 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-16T20:42:43.713 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-16T20:42:43.713 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-16T20:42:43.713 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-16T20:42:43.713 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-16T20:42:43.713 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-16T20:42:43.713 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply-3.11-14.el9.noarch 2023-12-16T20:42:43.713 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-16T20:42:43.713 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-16T20:42:43.713 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-16T20:42:43.713 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-16T20:42:43.713 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-16T20:42:43.713 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-16T20:42:43.713 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-16T20:42:43.714 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-16T20:42:43.714 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-16T20:42:43.714 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-16T20:42:43.714 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-16T20:42:43.714 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:43.714 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:42:43.774 INFO:teuthology.orchestra.run.smithi052.stdout:(15/33): ceph-osd-17.2.7-125.g1f2f5510.el9.x86_ 20 MB/s | 15 MB 00:00 2023-12-16T20:42:43.833 INFO:teuthology.orchestra.run.smithi052.stdout:(16/33): ceph-mon-17.2.7-125.g1f2f5510.el9.x86_ 3.6 MB/s | 3.7 MB 00:01 2023-12-16T20:42:43.900 INFO:teuthology.orchestra.run.smithi052.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 158 kB/s | 43 kB 00:00 2023-12-16T20:42:43.958 INFO:teuthology.orchestra.run.smithi052.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2023-12-16T20:42:44.017 INFO:teuthology.orchestra.run.smithi052.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2023-12-16T20:42:44.042 INFO:teuthology.orchestra.run.smithi052.stdout:(20/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 643 kB/s | 172 kB 00:00 2023-12-16T20:42:44.055 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-base 2023-12-16T20:42:44.076 INFO:teuthology.orchestra.run.smithi052.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 396 kB/s | 23 kB 00:00 2023-12-16T20:42:44.101 INFO:teuthology.orchestra.run.smithi052.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2023-12-16T20:42:44.126 INFO:teuthology.orchestra.run.smithi052.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 231 kB/s | 19 kB 00:00 2023-12-16T20:42:44.151 INFO:teuthology.orchestra.run.smithi052.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2023-12-16T20:42:44.177 INFO:teuthology.orchestra.run.smithi052.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2023-12-16T20:42:44.202 INFO:teuthology.orchestra.run.smithi052.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2023-12-16T20:42:44.236 INFO:teuthology.orchestra.run.smithi052.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2023-12-16T20:42:44.261 INFO:teuthology.orchestra.run.smithi052.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2023-12-16T20:42:44.295 INFO:teuthology.orchestra.run.smithi052.stdout:(29/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2023-12-16T20:42:44.320 INFO:teuthology.orchestra.run.smithi052.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 428 kB/s | 36 kB 00:00 2023-12-16T20:42:44.362 INFO:teuthology.orchestra.run.smithi052.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2023-12-16T20:42:44.378 INFO:teuthology.orchestra.run.smithi052.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2023-12-16T20:42:44.412 INFO:teuthology.orchestra.run.smithi052.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2023-12-16T20:42:44.416 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:42:44.416 INFO:teuthology.orchestra.run.smithi052.stdout:Total 14 MB/s | 27 MB 00:01 2023-12-16T20:42:44.416 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:42:44.459 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:42:44.459 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:42:44.614 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:52 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:42:44.707 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:42:44.756 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:42:44.757 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-12-16T20:42:44.757 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:42:44.816 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:42:44.816 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:42:44.825 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install cephadm 2023-12-16T20:42:45.380 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:53 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:42:45.522 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:42:45.522 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:42:45.522 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-12-16T20:42:45.522 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:42:45.523 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:42:45.523 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 80 k 2023-12-16T20:42:45.523 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:45.523 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:42:45.523 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:42:45.523 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-12-16T20:42:45.523 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:45.523 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 80 k 2023-12-16T20:42:45.523 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 364 k 2023-12-16T20:42:45.523 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:42:45.728 INFO:teuthology.orchestra.run.smithi150.stdout:cephadm-17.2.7-125.g1f2f5510.el9.noarch.rpm 393 kB/s | 80 kB 00:00 2023-12-16T20:42:45.729 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:42:45.729 INFO:teuthology.orchestra.run.smithi150.stdout:Total 390 kB/s | 80 kB 00:00 2023-12-16T20:42:45.729 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:42:45.734 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:42:45.734 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:42:45.736 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:42:45.736 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:42:45.804 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:42:46.067 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:42:46.132 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-12-16T20:42:46.165 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-12-16T20:42:46.198 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-12-16T20:42:46.224 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T20:42:46.294 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-12-16T20:42:46.333 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T20:42:46.370 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-12-16T20:42:46.417 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-12-16T20:42:46.491 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-12-16T20:42:46.551 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/33 2023-12-16T20:42:46.597 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2023-12-16T20:42:46.656 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2023-12-16T20:42:46.728 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2023-12-16T20:42:46.730 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T20:42:46.926 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2023-12-16T20:42:46.977 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2023-12-16T20:42:47.023 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2023-12-16T20:42:47.026 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T20:42:47.026 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:47.026 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:42:47.027 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:42:47.027 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:47.027 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:42:47.100 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2023-12-16T20:42:47.129 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2023-12-16T20:42:47.208 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2023-12-16T20:42:47.239 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-16T20:42:47.267 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-immutable-object-cache 2023-12-16T20:42:47.273 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2023-12-16T20:42:47.339 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2023-12-16T20:42:47.382 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2023-12-16T20:42:47.416 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-12-16T20:42:47.509 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-12-16T20:42:47.555 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-12-16T20:42:47.630 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-12-16T20:42:47.771 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-12-16T20:42:47.834 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:55 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:42:47.894 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-16T20:42:47.978 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:42:47.978 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:42:47.978 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-12-16T20:42:47.979 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:42:47.979 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:42:47.979 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 137 k 2023-12-16T20:42:47.979 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:47.979 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:42:47.979 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:42:47.979 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-12-16T20:42:47.979 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:47.979 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 137 k 2023-12-16T20:42:47.980 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 379 k 2023-12-16T20:42:47.980 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:42:48.061 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-125.g1f2f5510.el9.n 28/33 2023-12-16T20:42:48.092 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 29/33 2023-12-16T20:42:48.119 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 29/33 2023-12-16T20:42:48.119 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T20:42:48.120 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-16T20:42:48.120 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T20:42:48.120 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T20:42:48.120 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:48.201 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-immutable-object-cache-17.2.7-125.g1f2f551 620 kB/s | 137 kB 00:00 2023-12-16T20:42:48.202 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:42:48.202 INFO:teuthology.orchestra.run.smithi150.stdout:Total 616 kB/s | 137 kB 00:00 2023-12-16T20:42:48.202 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:42:48.209 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:42:48.209 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:42:48.261 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:42:48.262 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:42:48.396 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:42:48.460 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.e 1/1 2023-12-16T20:42:48.485 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.e 1/1 2023-12-16T20:42:48.485 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T20:42:48.485 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-16T20:42:48.485 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:49.490 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.e 1/1 2023-12-16T20:42:49.490 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:49.490 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:42:49.490 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:49.490 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:49.490 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:42:49.535 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 30/33 2023-12-16T20:42:49.563 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 30/33 2023-12-16T20:42:49.563 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T20:42:49.563 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-16T20:42:49.563 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T20:42:49.563 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T20:42:49.563 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:49.729 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr 2023-12-16T20:42:49.961 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 31/33 2023-12-16T20:42:49.985 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 31/33 2023-12-16T20:42:49.985 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T20:42:49.986 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-16T20:42:49.986 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T20:42:49.986 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T20:42:49.986 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:50.197 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 32/33 2023-12-16T20:42:50.219 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 32/33 2023-12-16T20:42:50.219 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T20:42:50.219 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-16T20:42:50.219 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T20:42:50.219 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T20:42:50.219 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:50.285 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 33/33 2023-12-16T20:42:50.296 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:58 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:42:50.390 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:42:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:42:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-12-16T20:42:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:42:50.509 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-dashboard 2023-12-16T20:42:51.063 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:59 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:42:51.211 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:42:51.212 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:42:51.212 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-12-16T20:42:51.212 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:42:51.212 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:42:51.212 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 1.7 M 2023-12-16T20:42:51.212 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-12-16T20:42:51.212 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 23 k 2023-12-16T20:42:51.212 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 15 k 2023-12-16T20:42:51.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-16T20:42:51.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-16T20:42:51.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-16T20:42:51.212 INFO:teuthology.orchestra.run.smithi150.stdout:Installing weak dependencies: 2023-12-16T20:42:51.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-16T20:42:51.212 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:51.213 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:42:51.213 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:42:51.213 INFO:teuthology.orchestra.run.smithi150.stdout:Install 7 Packages 2023-12-16T20:42:51.213 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:51.213 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.0 M 2023-12-16T20:42:51.213 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 8.1 M 2023-12-16T20:42:51.213 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:42:51.496 INFO:teuthology.orchestra.run.smithi150.stdout:(1/7): ceph-prometheus-alerts-17.2.7-125.g1f2f5 77 kB/s | 15 kB 00:00 2023-12-16T20:42:51.513 INFO:teuthology.orchestra.run.smithi150.stdout:(2/7): ceph-grafana-dashboards-17.2.7-125.g1f2f 112 kB/s | 23 kB 00:00 2023-12-16T20:42:51.688 INFO:teuthology.orchestra.run.smithi150.stdout:(3/7): ceph-mgr-dashboard-17.2.7-125.g1f2f5510. 4.4 MB/s | 1.7 MB 00:00 2023-12-16T20:42:51.705 INFO:teuthology.orchestra.run.smithi150.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 43 kB/s | 9.0 kB 00:00 2023-12-16T20:42:51.831 INFO:teuthology.orchestra.run.smithi150.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 129 kB/s | 41 kB 00:00 2023-12-16T20:42:51.939 INFO:teuthology.orchestra.run.smithi150.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 123 kB/s | 31 kB 00:00 2023-12-16T20:42:51.964 INFO:teuthology.orchestra.run.smithi150.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 729 kB/s | 188 kB 00:00 2023-12-16T20:42:51.968 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:42:51.968 INFO:teuthology.orchestra.run.smithi150.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2023-12-16T20:42:51.969 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:42:51.976 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:42:51.976 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:42:52.046 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:42:52.046 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:42:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 33/33 2023-12-16T20:42:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 1/33 2023-12-16T20:42:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 2/33 2023-12-16T20:42:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 3/33 2023-12-16T20:42:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 4/33 2023-12-16T20:42:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 5/33 2023-12-16T20:42:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-125.g1f2f5510.el9.n 6/33 2023-12-16T20:42:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-12-16T20:42:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-12-16T20:42:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2023-12-16T20:42:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2023-12-16T20:42:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2023-12-16T20:42:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2023-12-16T20:42:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2023-12-16T20:42:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2023-12-16T20:42:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2023-12-16T20:42:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 16/33 2023-12-16T20:42:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-12-16T20:42:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-16T20:42:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-12-16T20:42:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-12-16T20:42:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-12-16T20:42:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-12-16T20:42:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-12-16T20:42:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-12-16T20:42:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-12-16T20:42:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-12-16T20:42:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-16T20:42:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-12-16T20:42:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-12-16T20:42:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-12-16T20:42:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-12-16T20:42:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-12-16T20:42:52.353 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:42:52.416 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-16T20:42:52.450 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-16T20:42:52.505 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-16T20:42:52.536 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-16T20:42:52.565 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-125.g1f2f5510.el9.no 5/7 2023-12-16T20:42:52.797 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-125.g1f2f5510.el9.n 6/7 2023-12-16T20:42:52.826 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 7/7 2023-12-16T20:42:53.085 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-12-16T20:42:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:53.086 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:42:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:42:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-16T20:42:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-16T20:42:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-16T20:42:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-16T20:42:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-16T20:42:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply-3.11-14.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-16T20:42:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-16T20:42:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-16T20:42:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-16T20:42:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-16T20:42:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-16T20:42:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:53.088 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:42:53.380 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 7/7 2023-12-16T20:42:53.381 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-125.g1f2f5510.el9.n 1/7 2023-12-16T20:42:53.381 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 2/7 2023-12-16T20:42:53.381 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-125.g1f2f5510.el9.no 3/7 2023-12-16T20:42:53.381 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-16T20:42:53.381 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-16T20:42:53.381 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-16T20:42:53.455 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base 2023-12-16T20:42:53.710 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-16T20:42:53.710 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:53.710 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:42:53.710 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:42:53.710 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:42:53.710 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:42:53.710 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-16T20:42:53.710 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-16T20:42:53.710 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-16T20:42:53.711 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-16T20:42:53.711 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:53.711 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:42:53.908 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-16T20:42:54.035 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:52 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:42:54.131 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:42:54.183 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:42:54.184 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-12-16T20:42:54.184 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:42:54.250 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install cephadm 2023-12-16T20:42:54.467 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:02 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:42:54.611 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:42:54.612 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================== 2023-12-16T20:42:54.612 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-12-16T20:42:54.613 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================== 2023-12-16T20:42:54.613 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:42:54.613 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 7.4 M 2023-12-16T20:42:54.613 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-12-16T20:42:54.613 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-16T20:42:54.613 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-16T20:42:54.613 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-16T20:42:54.613 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-16T20:42:54.613 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-16T20:42:54.613 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-16T20:42:54.613 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-16T20:42:54.614 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-16T20:42:54.614 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-16T20:42:54.614 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-16T20:42:54.614 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-16T20:42:54.614 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:54.614 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:42:54.614 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================== 2023-12-16T20:42:54.614 INFO:teuthology.orchestra.run.smithi150.stdout:Install 12 Packages 2023-12-16T20:42:54.614 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:42:54.615 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 37 M 2023-12-16T20:42:54.615 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 209 M 2023-12-16T20:42:54.615 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:42:54.828 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:52 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:42:54.977 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:42:54.978 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:42:54.978 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-12-16T20:42:54.978 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:42:54.978 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:42:54.978 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 80 k 2023-12-16T20:42:54.978 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:54.978 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:42:54.978 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:42:54.978 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-12-16T20:42:54.978 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:54.978 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 80 k 2023-12-16T20:42:54.978 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 364 k 2023-12-16T20:42:54.978 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:42:55.029 INFO:teuthology.orchestra.run.smithi150.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 206 kB/s | 32 kB 00:00 2023-12-16T20:42:55.063 INFO:teuthology.orchestra.run.smithi150.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 525 kB/s | 17 kB 00:00 2023-12-16T20:42:55.121 INFO:teuthology.orchestra.run.smithi150.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-12-16T20:42:55.147 INFO:teuthology.orchestra.run.smithi150.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 421 kB/s | 34 kB 00:00 2023-12-16T20:42:55.210 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm-17.2.7-125.g1f2f5510.el9.noarch.rpm 348 kB/s | 80 kB 00:00 2023-12-16T20:42:55.210 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:42:55.211 INFO:teuthology.orchestra.run.smithi052.stdout:Total 346 kB/s | 80 kB 00:00 2023-12-16T20:42:55.211 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:42:55.216 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:42:55.216 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:42:55.218 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:42:55.219 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:42:55.264 INFO:teuthology.orchestra.run.smithi150.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 33 MB/s | 4.7 MB 00:00 2023-12-16T20:42:55.302 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:42:55.330 INFO:teuthology.orchestra.run.smithi150.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2023-12-16T20:42:55.356 INFO:teuthology.orchestra.run.smithi150.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2023-12-16T20:42:55.406 INFO:teuthology.orchestra.run.smithi150.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 16 MB/s | 803 kB 00:00 2023-12-16T20:42:55.565 INFO:teuthology.orchestra.run.smithi150.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 67 MB/s | 16 MB 00:00 2023-12-16T20:42:55.590 INFO:teuthology.orchestra.run.smithi150.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2023-12-16T20:42:55.690 INFO:teuthology.orchestra.run.smithi150.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-1 9.1 MB/s | 7.4 MB 00:00 2023-12-16T20:42:55.754 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T20:42:55.892 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T20:42:56.215 INFO:teuthology.orchestra.run.smithi150.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 378 kB/s | 245 kB 00:00 2023-12-16T20:42:56.219 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:42:56.219 INFO:teuthology.orchestra.run.smithi150.stdout:Total 23 MB/s | 37 MB 00:01 2023-12-16T20:42:56.232 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:42:56.257 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:42:56.257 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:42:56.271 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T20:42:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T20:42:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:56.578 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:42:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:42:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:56.578 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:42:56.695 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:42:56.695 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:42:56.783 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-immutable-object-cache 2023-12-16T20:42:57.362 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:55 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:42:57.395 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:42:57.464 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-16T20:42:57.495 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-16T20:42:57.514 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:42:57.514 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:42:57.514 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-12-16T20:42:57.515 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:42:57.515 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:42:57.515 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 137 k 2023-12-16T20:42:57.515 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:57.515 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:42:57.515 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:42:57.515 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-12-16T20:42:57.515 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:57.515 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 137 k 2023-12-16T20:42:57.515 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 379 k 2023-12-16T20:42:57.515 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:42:57.591 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-16T20:42:57.623 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-16T20:42:57.740 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-immutable-object-cache-17.2.7-125.g1f2f551 610 kB/s | 137 kB 00:00 2023-12-16T20:42:57.741 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:42:57.741 INFO:teuthology.orchestra.run.smithi052.stdout:Total 607 kB/s | 137 kB 00:00 2023-12-16T20:42:57.741 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:42:57.749 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:42:57.749 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:42:57.800 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-16T20:42:57.803 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:42:57.804 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:42:57.830 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-16T20:42:57.913 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-16T20:42:57.947 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:42:58.016 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.e 1/1 2023-12-16T20:42:58.042 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.e 1/1 2023-12-16T20:42:58.042 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T20:42:58.043 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-16T20:42:58.043 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:58.585 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-16T20:42:58.684 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-16T20:42:59.176 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.e 1/1 2023-12-16T20:42:59.177 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:59.177 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:42:59.177 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:42:59.177 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:42:59.177 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:42:59.389 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr 2023-12-16T20:42:59.940 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-16T20:42:59.976 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:57 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:00.072 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:43:00.124 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:00.125 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-12-16T20:43:00.125 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:00.188 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-dashboard 2023-12-16T20:43:00.763 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:58 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:00.913 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:00.917 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:00.917 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-12-16T20:43:00.917 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:00.917 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:43:00.917 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 1.7 M 2023-12-16T20:43:00.917 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-12-16T20:43:00.917 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 23 k 2023-12-16T20:43:00.918 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 15 k 2023-12-16T20:43:00.918 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-16T20:43:00.918 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-16T20:43:00.918 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-16T20:43:00.918 INFO:teuthology.orchestra.run.smithi052.stdout:Installing weak dependencies: 2023-12-16T20:43:00.918 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-16T20:43:00.918 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:00.918 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:43:00.918 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:00.918 INFO:teuthology.orchestra.run.smithi052.stdout:Install 7 Packages 2023-12-16T20:43:00.918 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:00.918 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 2.0 M 2023-12-16T20:43:00.918 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 8.1 M 2023-12-16T20:43:00.918 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:43:00.958 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-16T20:43:00.988 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f55 12/12 2023-12-16T20:43:01.276 INFO:teuthology.orchestra.run.smithi052.stdout:(1/7): ceph-prometheus-alerts-17.2.7-125.g1f2f5 70 kB/s | 15 kB 00:00 2023-12-16T20:43:01.293 INFO:teuthology.orchestra.run.smithi052.stdout:(2/7): ceph-grafana-dashboards-17.2.7-125.g1f2f 104 kB/s | 23 kB 00:00 2023-12-16T20:43:01.335 INFO:teuthology.orchestra.run.smithi052.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 155 kB/s | 9.0 kB 00:00 2023-12-16T20:43:01.368 INFO:teuthology.orchestra.run.smithi052.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 543 kB/s | 41 kB 00:00 2023-12-16T20:43:01.385 INFO:teuthology.orchestra.run.smithi052.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 612 kB/s | 31 kB 00:00 2023-12-16T20:43:01.452 INFO:teuthology.orchestra.run.smithi052.stdout:(6/7): ceph-mgr-dashboard-17.2.7-125.g1f2f5510. 4.4 MB/s | 1.7 MB 00:00 2023-12-16T20:43:01.478 INFO:teuthology.orchestra.run.smithi052.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2023-12-16T20:43:01.479 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:01.480 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.5 MB/s | 2.0 MB 00:00 2023-12-16T20:43:01.480 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:43:01.488 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:43:01.488 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:43:01.561 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:43:01.562 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:43:01.873 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:43:01.937 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-16T20:43:01.971 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-16T20:43:02.028 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-16T20:43:02.058 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-16T20:43:02.087 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-125.g1f2f5510.el9.no 5/7 2023-12-16T20:43:02.329 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-125.g1f2f5510.el9.n 6/7 2023-12-16T20:43:02.364 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 7/7 2023-12-16T20:43:03.047 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 7/7 2023-12-16T20:43:03.048 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-125.g1f2f5510.el9.n 1/7 2023-12-16T20:43:03.048 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 2/7 2023-12-16T20:43:03.048 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-125.g1f2f5510.el9.no 3/7 2023-12-16T20:43:03.048 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-16T20:43:03.048 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-16T20:43:03.048 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-16T20:43:03.369 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-16T20:43:03.370 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:03.370 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:43:03.370 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:43:03.370 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:43:03.370 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:43:03.370 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-16T20:43:03.370 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-16T20:43:03.370 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-16T20:43:03.370 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-16T20:43:03.370 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:03.370 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:03.592 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-16T20:43:03.594 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f55 12/12 2023-12-16T20:43:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f55 1/12 2023-12-16T20:43:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-16T20:43:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-16T20:43:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-16T20:43:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-16T20:43:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-16T20:43:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-16T20:43:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-16T20:43:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-16T20:43:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-16T20:43:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-16T20:43:04.062 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-16T20:43:04.062 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:04.062 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:43:04.062 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:43:04.062 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-16T20:43:04.062 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-16T20:43:04.062 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-16T20:43:04.062 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-16T20:43:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-16T20:43:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-16T20:43:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-16T20:43:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-16T20:43:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-16T20:43:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-16T20:43:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-16T20:43:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:04.063 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:04.176 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:02 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:04.328 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:04.329 INFO:teuthology.orchestra.run.smithi052.stdout:=========================================================================================== 2023-12-16T20:43:04.329 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-12-16T20:43:04.329 INFO:teuthology.orchestra.run.smithi052.stdout:=========================================================================================== 2023-12-16T20:43:04.329 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:43:04.330 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 7.4 M 2023-12-16T20:43:04.330 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-12-16T20:43:04.330 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-16T20:43:04.330 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-16T20:43:04.330 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-16T20:43:04.330 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-16T20:43:04.330 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-16T20:43:04.330 INFO:teuthology.orchestra.run.smithi052.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-16T20:43:04.330 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-16T20:43:04.330 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-16T20:43:04.330 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-16T20:43:04.330 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-16T20:43:04.330 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-16T20:43:04.330 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:04.330 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:43:04.331 INFO:teuthology.orchestra.run.smithi052.stdout:=========================================================================================== 2023-12-16T20:43:04.331 INFO:teuthology.orchestra.run.smithi052.stdout:Install 12 Packages 2023-12-16T20:43:04.331 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:04.331 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 37 M 2023-12-16T20:43:04.331 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 209 M 2023-12-16T20:43:04.331 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:43:04.416 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-rook 2023-12-16T20:43:04.543 INFO:teuthology.orchestra.run.smithi052.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 211 kB/s | 32 kB 00:00 2023-12-16T20:43:04.568 INFO:teuthology.orchestra.run.smithi052.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 702 kB/s | 17 kB 00:00 2023-12-16T20:43:04.644 INFO:teuthology.orchestra.run.smithi052.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-12-16T20:43:04.660 INFO:teuthology.orchestra.run.smithi052.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 378 kB/s | 34 kB 00:00 2023-12-16T20:43:04.794 INFO:teuthology.orchestra.run.smithi052.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 36 MB/s | 4.7 MB 00:00 2023-12-16T20:43:04.861 INFO:teuthology.orchestra.run.smithi052.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2023-12-16T20:43:04.894 INFO:teuthology.orchestra.run.smithi052.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2023-12-16T20:43:04.953 INFO:teuthology.orchestra.run.smithi052.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 14 MB/s | 803 kB 00:00 2023-12-16T20:43:04.977 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:12 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:43:05.120 INFO:teuthology.orchestra.run.smithi052.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 61 MB/s | 16 MB 00:00 2023-12-16T20:43:05.122 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:43:05.124 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2023-12-16T20:43:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-12-16T20:43:05.124 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2023-12-16T20:43:05.124 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:43:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 47 k 2023-12-16T20:43:05.124 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-12-16T20:43:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-16T20:43:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-16T20:43:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-16T20:43:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-16T20:43:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-16T20:43:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-16T20:43:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-16T20:43:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-16T20:43:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-16T20:43:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-16T20:43:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-16T20:43:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-16T20:43:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:05.125 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:43:05.125 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2023-12-16T20:43:05.125 INFO:teuthology.orchestra.run.smithi150.stdout:Install 13 Packages 2023-12-16T20:43:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:05.126 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.0 M 2023-12-16T20:43:05.126 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 26 M 2023-12-16T20:43:05.126 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:43:05.145 INFO:teuthology.orchestra.run.smithi052.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2023-12-16T20:43:05.246 INFO:teuthology.orchestra.run.smithi052.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-1 8.7 MB/s | 7.4 MB 00:00 2023-12-16T20:43:05.371 INFO:teuthology.orchestra.run.smithi052.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 981 kB/s | 245 kB 00:00 2023-12-16T20:43:05.373 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:05.373 INFO:teuthology.orchestra.run.smithi052.stdout:Total 36 MB/s | 37 MB 00:01 2023-12-16T20:43:05.387 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:43:05.404 INFO:teuthology.orchestra.run.smithi150.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 145 kB/s | 18 kB 00:00 2023-12-16T20:43:05.414 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:43:05.414 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:43:05.429 INFO:teuthology.orchestra.run.smithi150.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 162 kB/s | 24 kB 00:00 2023-12-16T20:43:05.471 INFO:teuthology.orchestra.run.smithi150.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-12-16T20:43:05.497 INFO:teuthology.orchestra.run.smithi150.stdout:(4/13): ceph-mgr-rook-17.2.7-125.g1f2f5510.el9. 218 kB/s | 47 kB 00:00 2023-12-16T20:43:05.522 INFO:teuthology.orchestra.run.smithi150.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2023-12-16T20:43:05.547 INFO:teuthology.orchestra.run.smithi150.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2023-12-16T20:43:05.572 INFO:teuthology.orchestra.run.smithi150.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 860 kB/s | 43 kB 00:00 2023-12-16T20:43:05.756 INFO:teuthology.orchestra.run.smithi150.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 72 kB/s | 15 kB 00:00 2023-12-16T20:43:05.848 INFO:teuthology.orchestra.run.smithi150.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 99 kB/s | 32 kB 00:00 2023-12-16T20:43:05.888 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:43:05.924 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:43:05.990 INFO:teuthology.orchestra.run.smithi150.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 417 kB/s | 59 kB 00:00 2023-12-16T20:43:06.015 INFO:teuthology.orchestra.run.smithi150.stdout:(11/13): python3-google-auth-2.23.4-1.el9.noarc 446 kB/s | 197 kB 00:00 2023-12-16T20:43:06.099 INFO:teuthology.orchestra.run.smithi150.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 827 kB/s | 90 kB 00:00 2023-12-16T20:43:06.132 INFO:teuthology.orchestra.run.smithi150.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.7 MB/s | 1.0 MB 00:00 2023-12-16T20:43:06.137 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:06.137 INFO:teuthology.orchestra.run.smithi150.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2023-12-16T20:43:06.137 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:43:06.150 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:43:06.150 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:43:06.250 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:43:06.250 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:43:06.747 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:43:06.775 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:43:06.835 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-16T20:43:06.875 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-16T20:43:06.940 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-16T20:43:06.974 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-16T20:43:06.992 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-16T20:43:07.012 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-16T20:43:07.047 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-16T20:43:07.084 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-16T20:43:07.126 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-16T20:43:07.207 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-16T20:43:07.220 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-16T20:43:07.236 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-16T20:43:07.322 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-16T20:43:07.338 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-16T20:43:07.386 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-16T20:43:08.008 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-16T20:43:08.115 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-16T20:43:08.136 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-16T20:43:08.166 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-16T20:43:08.199 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-16T20:43:08.237 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-16T20:43:08.261 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 13/13 2023-12-16T20:43:09.056 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 13/13 2023-12-16T20:43:09.056 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 1/13 2023-12-16T20:43:09.056 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-16T20:43:09.056 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-16T20:43:09.057 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-16T20:43:09.057 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-16T20:43:09.057 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-16T20:43:09.057 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-16T20:43:09.057 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-16T20:43:09.057 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-16T20:43:09.057 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-16T20:43:09.057 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-16T20:43:09.057 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-16T20:43:09.442 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-16T20:43:09.442 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:09.443 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:43:09.443 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:43:09.443 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-16T20:43:09.443 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-16T20:43:09.443 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-16T20:43:09.443 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-16T20:43:09.443 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-16T20:43:09.443 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-16T20:43:09.443 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-16T20:43:09.443 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-16T20:43:09.443 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-16T20:43:09.443 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-16T20:43:09.443 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-16T20:43:09.443 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-16T20:43:09.443 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:09.444 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:09.461 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-16T20:43:09.683 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-cephadm 2023-12-16T20:43:10.258 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:18 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:43:10.412 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:43:10.413 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2023-12-16T20:43:10.413 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-12-16T20:43:10.413 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2023-12-16T20:43:10.414 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:43:10.414 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 121 k 2023-12-16T20:43:10.414 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-12-16T20:43:10.414 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-16T20:43:10.414 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-16T20:43:10.414 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-16T20:43:10.414 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-16T20:43:10.414 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-16T20:43:10.414 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:10.414 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:43:10.414 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2023-12-16T20:43:10.414 INFO:teuthology.orchestra.run.smithi150.stdout:Install 6 Packages 2023-12-16T20:43:10.414 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:10.415 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 6.8 M 2023-12-16T20:43:10.415 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 33 M 2023-12-16T20:43:10.415 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:43:10.538 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-16T20:43:10.567 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f55 12/12 2023-12-16T20:43:10.617 INFO:teuthology.orchestra.run.smithi150.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-12-16T20:43:10.742 INFO:teuthology.orchestra.run.smithi150.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2023-12-16T20:43:10.784 INFO:teuthology.orchestra.run.smithi150.stdout:(3/6): ceph-mgr-cephadm-17.2.7-125.g1f2f5510.el 381 kB/s | 121 kB 00:00 2023-12-16T20:43:10.851 INFO:teuthology.orchestra.run.smithi150.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 2.3 MB/s | 548 kB 00:00 2023-12-16T20:43:10.868 INFO:teuthology.orchestra.run.smithi150.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 461 kB/s | 58 kB 00:00 2023-12-16T20:43:10.885 INFO:teuthology.orchestra.run.smithi150.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 509 kB/s | 51 kB 00:00 2023-12-16T20:43:10.888 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:10.888 INFO:teuthology.orchestra.run.smithi150.stdout:Total 14 MB/s | 6.8 MB 00:00 2023-12-16T20:43:10.888 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:43:10.902 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:43:10.903 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:43:11.034 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:43:11.035 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:43:11.336 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:43:11.491 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-16T20:43:11.544 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-16T20:43:12.052 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-16T20:43:12.137 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-16T20:43:12.193 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-16T20:43:12.216 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 6/6 2023-12-16T20:43:13.100 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 6/6 2023-12-16T20:43:13.101 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/6 2023-12-16T20:43:13.101 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-16T20:43:13.101 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-16T20:43:13.101 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-16T20:43:13.101 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-16T20:43:13.228 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f55 12/12 2023-12-16T20:43:13.228 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f55 1/12 2023-12-16T20:43:13.228 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-16T20:43:13.228 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-16T20:43:13.228 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-16T20:43:13.228 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-16T20:43:13.228 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-16T20:43:13.228 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-16T20:43:13.228 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-16T20:43:13.228 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-16T20:43:13.228 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-16T20:43:13.229 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-16T20:43:13.416 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-16T20:43:13.416 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:13.416 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:43:13.416 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:43:13.416 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-16T20:43:13.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-16T20:43:13.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-16T20:43:13.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-16T20:43:13.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-16T20:43:13.417 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:13.417 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:13.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-16T20:43:13.655 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:13.655 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:43:13.655 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:43:13.655 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-16T20:43:13.655 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-16T20:43:13.655 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-16T20:43:13.655 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-16T20:43:13.655 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-16T20:43:13.655 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-16T20:43:13.656 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-16T20:43:13.656 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-16T20:43:13.656 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-16T20:43:13.656 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-16T20:43:13.656 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-16T20:43:13.656 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:13.656 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:13.663 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-fuse 2023-12-16T20:43:14.005 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-rook 2023-12-16T20:43:14.221 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:22 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:43:14.364 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:43:14.365 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:14.365 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-12-16T20:43:14.365 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:14.366 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:43:14.366 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 810 k 2023-12-16T20:43:14.366 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-12-16T20:43:14.366 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-16T20:43:14.366 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:14.366 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:43:14.366 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:14.366 INFO:teuthology.orchestra.run.smithi150.stdout:Install 2 Packages 2023-12-16T20:43:14.366 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:14.367 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 889 k 2023-12-16T20:43:14.367 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 2.6 M 2023-12-16T20:43:14.367 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:43:14.489 INFO:teuthology.orchestra.run.smithi150.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 645 kB/s | 79 kB 00:00 2023-12-16T20:43:14.597 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:12 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:14.752 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:14.753 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2023-12-16T20:43:14.753 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-12-16T20:43:14.753 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2023-12-16T20:43:14.753 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:43:14.753 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 47 k 2023-12-16T20:43:14.753 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-12-16T20:43:14.753 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-16T20:43:14.753 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-16T20:43:14.753 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-16T20:43:14.754 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-16T20:43:14.754 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-16T20:43:14.754 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-16T20:43:14.754 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-16T20:43:14.754 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-16T20:43:14.754 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-16T20:43:14.754 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-16T20:43:14.754 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-16T20:43:14.754 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-16T20:43:14.754 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:43:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2023-12-16T20:43:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:Install 13 Packages 2023-12-16T20:43:14.754 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:14.755 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 2.0 M 2023-12-16T20:43:14.755 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 26 M 2023-12-16T20:43:14.755 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:43:14.805 INFO:teuthology.orchestra.run.smithi150.stdout:(2/2): ceph-fuse-17.2.7-125.g1f2f5510.el9.x86_6 1.8 MB/s | 810 kB 00:00 2023-12-16T20:43:14.806 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:14.807 INFO:teuthology.orchestra.run.smithi150.stdout:Total 2.0 MB/s | 889 kB 00:00 2023-12-16T20:43:14.807 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:43:14.826 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:43:14.827 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:43:14.900 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:43:14.901 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:43:15.041 INFO:teuthology.orchestra.run.smithi052.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 142 kB/s | 18 kB 00:00 2023-12-16T20:43:15.098 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:43:15.183 INFO:teuthology.orchestra.run.smithi052.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 91 kB/s | 24 kB 00:00 2023-12-16T20:43:15.225 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-16T20:43:15.242 INFO:teuthology.orchestra.run.smithi052.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 877 kB/s | 174 kB 00:00 2023-12-16T20:43:15.267 INFO:teuthology.orchestra.run.smithi052.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2023-12-16T20:43:15.288 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 2/2 2023-12-16T20:43:15.292 INFO:teuthology.orchestra.run.smithi052.stdout:(5/13): ceph-mgr-rook-17.2.7-125.g1f2f5510.el9. 125 kB/s | 47 kB 00:00 2023-12-16T20:43:15.318 INFO:teuthology.orchestra.run.smithi052.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2023-12-16T20:43:15.343 INFO:teuthology.orchestra.run.smithi052.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 568 kB/s | 43 kB 00:00 2023-12-16T20:43:15.560 INFO:teuthology.orchestra.run.smithi052.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 121 kB/s | 32 kB 00:00 2023-12-16T20:43:15.585 INFO:teuthology.orchestra.run.smithi052.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 62 kB/s | 15 kB 00:00 2023-12-16T20:43:15.611 INFO:teuthology.orchestra.run.smithi052.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 739 kB/s | 197 kB 00:00 2023-12-16T20:43:15.670 INFO:teuthology.orchestra.run.smithi052.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 710 kB/s | 59 kB 00:00 2023-12-16T20:43:15.694 INFO:teuthology.orchestra.run.smithi052.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2023-12-16T20:43:15.728 INFO:teuthology.orchestra.run.smithi052.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2023-12-16T20:43:15.733 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:15.733 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2023-12-16T20:43:15.733 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:43:15.747 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:43:15.747 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:43:15.853 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:43:15.853 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:43:15.986 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 2/2 2023-12-16T20:43:15.987 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/2 2023-12-16T20:43:16.299 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-16T20:43:16.299 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:16.300 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:43:16.300 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-16T20:43:16.300 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:16.300 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:16.382 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:43:16.532 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados-devel 2023-12-16T20:43:16.568 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-16T20:43:16.631 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-16T20:43:16.686 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-16T20:43:16.722 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-16T20:43:16.765 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-16T20:43:16.860 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-16T20:43:16.972 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-16T20:43:17.033 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-16T20:43:17.096 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:25 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:43:17.245 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:43:17.246 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:17.246 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-12-16T20:43:17.246 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:17.246 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:43:17.246 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 126 k 2023-12-16T20:43:17.246 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:17.246 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:43:17.246 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:17.246 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-12-16T20:43:17.246 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:17.246 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 126 k 2023-12-16T20:43:17.246 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 448 k 2023-12-16T20:43:17.246 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:43:17.559 INFO:teuthology.orchestra.run.smithi150.stdout:librados-devel-17.2.7-125.g1f2f5510.el9.x86_64. 403 kB/s | 126 kB 00:00 2023-12-16T20:43:17.560 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:17.560 INFO:teuthology.orchestra.run.smithi150.stdout:Total 401 kB/s | 126 kB 00:00 2023-12-16T20:43:17.560 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:43:17.566 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:43:17.566 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:43:17.620 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:43:17.621 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:43:17.763 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:43:17.810 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-16T20:43:17.855 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-16T20:43:17.857 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:17.889 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-16T20:43:17.935 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-16T20:43:17.958 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 13/13 2023-12-16T20:43:18.302 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:18.621 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:18.621 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:18.621 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:43:18.622 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:43:18.622 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:18.622 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:18.803 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 13/13 2023-12-16T20:43:18.804 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 1/13 2023-12-16T20:43:18.804 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-16T20:43:18.804 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-16T20:43:18.804 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-16T20:43:18.804 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-16T20:43:18.804 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-16T20:43:18.804 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-16T20:43:18.804 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-16T20:43:18.804 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-16T20:43:18.804 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-16T20:43:18.804 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-16T20:43:18.804 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-16T20:43:18.853 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs2 2023-12-16T20:43:19.184 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-16T20:43:19.184 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:19.184 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:43:19.184 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:43:19.185 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-16T20:43:19.185 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-16T20:43:19.185 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-16T20:43:19.185 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-16T20:43:19.185 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-16T20:43:19.185 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-16T20:43:19.185 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-16T20:43:19.185 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-16T20:43:19.185 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-16T20:43:19.185 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-16T20:43:19.185 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-16T20:43:19.186 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-16T20:43:19.186 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:19.186 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:19.412 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-cephadm 2023-12-16T20:43:19.413 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:27 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:43:19.509 INFO:teuthology.orchestra.run.smithi150.stdout:Package libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:43:19.560 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:43:19.561 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-12-16T20:43:19.561 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:19.627 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs-devel 2023-12-16T20:43:19.990 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:17 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:20.140 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:20.140 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2023-12-16T20:43:20.140 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-12-16T20:43:20.141 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2023-12-16T20:43:20.141 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:43:20.141 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 121 k 2023-12-16T20:43:20.141 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-12-16T20:43:20.141 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-16T20:43:20.141 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-16T20:43:20.141 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-16T20:43:20.141 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-16T20:43:20.141 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-16T20:43:20.141 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:20.141 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:43:20.141 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2023-12-16T20:43:20.141 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2023-12-16T20:43:20.142 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:20.142 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 6.8 M 2023-12-16T20:43:20.142 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 33 M 2023-12-16T20:43:20.142 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:43:20.187 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:28 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:43:20.331 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:43:20.332 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:20.332 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-12-16T20:43:20.332 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:20.332 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:43:20.332 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 25 k 2023-12-16T20:43:20.332 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:20.332 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:43:20.333 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:20.333 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-12-16T20:43:20.333 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:20.333 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 25 k 2023-12-16T20:43:20.333 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 110 k 2023-12-16T20:43:20.333 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:43:20.364 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-12-16T20:43:20.464 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2023-12-16T20:43:20.513 INFO:teuthology.orchestra.run.smithi150.stdout:libcephfs-devel-17.2.7-125.g1f2f5510.el9.x86_64 139 kB/s | 25 kB 00:00 2023-12-16T20:43:20.513 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:20.513 INFO:teuthology.orchestra.run.smithi150.stdout:Total 138 kB/s | 25 kB 00:00 2023-12-16T20:43:20.513 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:43:20.517 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:43:20.517 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:43:20.518 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:43:20.518 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:43:20.540 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): ceph-mgr-cephadm-17.2.7-125.g1f2f5510.el 354 kB/s | 121 kB 00:00 2023-12-16T20:43:20.593 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:43:20.712 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:21.023 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 103 kB/s | 58 kB 00:00 2023-12-16T20:43:21.065 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 97 kB/s | 51 kB 00:00 2023-12-16T20:43:21.086 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:21.240 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 626 kB/s | 548 kB 00:00 2023-12-16T20:43:21.243 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:21.243 INFO:teuthology.orchestra.run.smithi052.stdout:Total 6.1 MB/s | 6.8 MB 00:01 2023-12-16T20:43:21.243 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:43:21.254 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:43:21.254 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:43:21.354 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:43:21.355 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:43:21.356 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:21.356 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:21.356 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:43:21.356 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:43:21.356 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:21.356 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:21.564 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados2 2023-12-16T20:43:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:43:21.840 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-16T20:43:21.893 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-16T20:43:22.123 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:30 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:43:22.218 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:43:22.267 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:43:22.268 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-12-16T20:43:22.268 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:22.325 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librbd1 2023-12-16T20:43:22.444 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-16T20:43:22.530 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-16T20:43:22.592 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-16T20:43:22.620 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 6/6 2023-12-16T20:43:22.886 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:30 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:43:22.980 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:43:23.030 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:43:23.031 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-12-16T20:43:23.031 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:23.092 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rados 2023-12-16T20:43:23.443 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 6/6 2023-12-16T20:43:23.443 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/6 2023-12-16T20:43:23.443 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-16T20:43:23.443 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-16T20:43:23.443 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-16T20:43:23.443 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-16T20:43:23.643 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:31 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:43:23.738 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:43:23.757 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-16T20:43:23.757 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:23.757 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:43:23.757 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T20:43:23.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-16T20:43:23.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-16T20:43:23.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-16T20:43:23.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-16T20:43:23.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-16T20:43:23.758 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:23.758 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:23.789 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:43:23.790 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-12-16T20:43:23.791 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:23.860 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rgw 2023-12-16T20:43:24.000 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-fuse 2023-12-16T20:43:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:32 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:43:24.507 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:43:24.557 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:43:24.558 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-12-16T20:43:24.558 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:24.585 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:22 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:24.630 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-cephfs 2023-12-16T20:43:24.736 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:24.737 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:24.737 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-12-16T20:43:24.737 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:24.737 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:43:24.737 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 810 k 2023-12-16T20:43:24.737 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-12-16T20:43:24.737 INFO:teuthology.orchestra.run.smithi052.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-16T20:43:24.737 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:24.737 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:43:24.737 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:24.737 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2023-12-16T20:43:24.738 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:24.738 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 889 k 2023-12-16T20:43:24.738 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.6 M 2023-12-16T20:43:24.738 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:43:24.866 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 616 kB/s | 79 kB 00:00 2023-12-16T20:43:25.188 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:33 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:43:25.200 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): ceph-fuse-17.2.7-125.g1f2f5510.el9.x86_6 1.7 MB/s | 810 kB 00:00 2023-12-16T20:43:25.201 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:25.202 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.9 MB/s | 889 kB 00:00 2023-12-16T20:43:25.202 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:43:25.223 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:43:25.223 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:43:25.282 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:43:25.302 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:43:25.303 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:43:25.333 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:43:25.334 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-12-16T20:43:25.334 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:25.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rbd 2023-12-16T20:43:25.502 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:43:25.641 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-16T20:43:25.710 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 2/2 2023-12-16T20:43:25.962 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:33 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:43:26.056 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:43:26.106 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:43:26.107 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-12-16T20:43:26.108 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:26.186 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-fuse 2023-12-16T20:43:26.403 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 2/2 2023-12-16T20:43:26.403 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/2 2023-12-16T20:43:26.725 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-16T20:43:26.725 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:26.725 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:43:26.725 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-16T20:43:26.725 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:26.725 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:26.744 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:34 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:43:26.889 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:43:26.889 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:26.889 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2023-12-16T20:43:26.889 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:26.889 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:43:26.889 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 81 k 2023-12-16T20:43:26.889 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:26.890 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:43:26.890 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:26.890 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-12-16T20:43:26.890 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:26.890 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 81 k 2023-12-16T20:43:26.890 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 222 k 2023-12-16T20:43:26.890 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:43:26.999 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados-devel 2023-12-16T20:43:27.233 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-fuse-17.2.7-125.g1f2f5510.el9.x86_64.rpm 238 kB/s | 81 kB 00:00 2023-12-16T20:43:27.234 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:27.234 INFO:teuthology.orchestra.run.smithi150.stdout:Total 237 kB/s | 81 kB 00:00 2023-12-16T20:43:27.234 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:43:27.241 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:43:27.241 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:43:27.296 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:43:27.296 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:43:27.440 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:43:27.530 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:27.587 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:25 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:27.739 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:27.740 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:27.740 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-12-16T20:43:27.740 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:27.740 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:43:27.740 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 126 k 2023-12-16T20:43:27.740 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:27.740 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:43:27.740 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:27.740 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-12-16T20:43:27.740 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:27.740 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 126 k 2023-12-16T20:43:27.740 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 448 k 2023-12-16T20:43:27.741 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:43:27.942 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:28.053 INFO:teuthology.orchestra.run.smithi052.stdout:librados-devel-17.2.7-125.g1f2f5510.el9.x86_64. 403 kB/s | 126 kB 00:00 2023-12-16T20:43:28.054 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:28.054 INFO:teuthology.orchestra.run.smithi052.stdout:Total 402 kB/s | 126 kB 00:00 2023-12-16T20:43:28.054 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:43:28.061 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:43:28.061 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:43:28.120 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:43:28.121 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:43:28.221 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:28.222 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:28.222 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:43:28.222 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:43:28.222 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:28.222 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:28.267 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:43:28.371 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:28.446 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-mirror 2023-12-16T20:43:28.817 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:29.012 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:37 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:43:29.158 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:29.158 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:29.158 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:43:29.158 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:43:29.158 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:29.158 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:29.161 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:43:29.162 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:29.162 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-12-16T20:43:29.162 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:29.162 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:43:29.162 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 2.9 M 2023-12-16T20:43:29.162 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:29.162 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:43:29.162 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:29.162 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-12-16T20:43:29.163 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:29.163 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.9 M 2023-12-16T20:43:29.163 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 12 M 2023-12-16T20:43:29.163 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:43:29.381 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs2 2023-12-16T20:43:29.736 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-mirror-17.2.7-125.g1f2f5510.el9.x86_64.rpm 5.1 MB/s | 2.9 MB 00:00 2023-12-16T20:43:29.736 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:29.737 INFO:teuthology.orchestra.run.smithi150.stdout:Total 5.1 MB/s | 2.9 MB 00:00 2023-12-16T20:43:29.737 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:43:29.745 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:43:29.745 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:43:29.816 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:43:29.816 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:43:29.963 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:27 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:30.060 INFO:teuthology.orchestra.run.smithi052.stdout:Package libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:43:30.114 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:30.114 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-12-16T20:43:30.115 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:30.180 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs-devel 2023-12-16T20:43:30.225 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:43:30.286 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:30.311 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:30.311 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T20:43:30.311 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-16T20:43:30.311 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T20:43:30.311 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T20:43:30.311 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:30.763 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:28 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:30.914 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:30.915 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:30.915 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-12-16T20:43:30.915 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:30.915 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:43:30.915 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 25 k 2023-12-16T20:43:30.915 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:30.915 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:43:30.915 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:30.915 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-12-16T20:43:30.915 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:30.915 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 25 k 2023-12-16T20:43:30.916 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 110 k 2023-12-16T20:43:30.916 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:43:31.073 INFO:teuthology.orchestra.run.smithi052.stdout:libcephfs-devel-17.2.7-125.g1f2f5510.el9.x86_64 159 kB/s | 25 kB 00:00 2023-12-16T20:43:31.074 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:31.074 INFO:teuthology.orchestra.run.smithi052.stdout:Total 158 kB/s | 25 kB 00:00 2023-12-16T20:43:31.074 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:43:31.077 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:43:31.078 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:43:31.079 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:43:31.079 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:43:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:43:31.281 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:31.491 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:31.491 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:31.491 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:43:31.491 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:43:31.491 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:31.491 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:31.665 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:31.763 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-nbd 2023-12-16T20:43:31.955 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:31.956 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:31.956 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:43:31.956 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:43:31.956 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:31.956 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:32.184 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados2 2023-12-16T20:43:32.338 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:40 ago on Sat 16 Dec 2023 08:41:52 PM UTC. 2023-12-16T20:43:32.481 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-12-16T20:43:32.482 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:32.482 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2023-12-16T20:43:32.482 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:32.482 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-12-16T20:43:32.483 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 162 k 2023-12-16T20:43:32.483 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:32.483 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-12-16T20:43:32.483 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-12-16T20:43:32.483 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-12-16T20:43:32.483 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:32.483 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 162 k 2023-12-16T20:43:32.483 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 465 k 2023-12-16T20:43:32.483 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-12-16T20:43:32.770 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:30 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:32.822 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-nbd-17.2.7-125.g1f2f5510.el9.x86_64.rpm 478 kB/s | 162 kB 00:00 2023-12-16T20:43:32.823 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:32.823 INFO:teuthology.orchestra.run.smithi150.stdout:Total 477 kB/s | 162 kB 00:00 2023-12-16T20:43:32.823 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-12-16T20:43:32.831 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-12-16T20:43:32.831 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-12-16T20:43:32.869 INFO:teuthology.orchestra.run.smithi052.stdout:Package librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:43:32.886 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-12-16T20:43:32.886 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-12-16T20:43:32.922 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:32.923 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-12-16T20:43:32.923 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:32.988 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librbd1 2023-12-16T20:43:33.050 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-12-16T20:43:33.145 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:33.575 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:31 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:33.599 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:33.674 INFO:teuthology.orchestra.run.smithi052.stdout:Package librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:43:33.727 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:33.728 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-12-16T20:43:33.728 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:33.800 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rados 2023-12-16T20:43:33.878 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:33.879 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:33.879 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-12-16T20:43:33.879 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:43:33.879 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:43:33.879 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-12-16T20:43:34.090 DEBUG:teuthology.parallel:result is None 2023-12-16T20:43:34.377 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:32 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:34.473 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:43:34.527 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:34.528 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-12-16T20:43:34.528 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:34.594 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rgw 2023-12-16T20:43:35.173 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:33 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:35.270 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:43:35.324 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:35.325 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-12-16T20:43:35.325 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:35.383 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-cephfs 2023-12-16T20:43:35.966 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:33 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:36.065 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:43:36.118 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:36.119 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-12-16T20:43:36.120 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:36.185 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rbd 2023-12-16T20:43:36.765 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:34 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:36.863 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T20:43:36.916 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:36.917 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-12-16T20:43:36.917 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:36.985 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-fuse 2023-12-16T20:43:37.576 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:35 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:37.729 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:37.729 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:37.729 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2023-12-16T20:43:37.730 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:37.730 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:43:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 81 k 2023-12-16T20:43:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:37.730 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:43:37.730 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:37.730 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-12-16T20:43:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:37.730 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 81 k 2023-12-16T20:43:37.730 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 222 k 2023-12-16T20:43:37.730 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:43:37.943 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-fuse-17.2.7-125.g1f2f5510.el9.x86_64.rpm 383 kB/s | 81 kB 00:00 2023-12-16T20:43:37.944 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:37.944 INFO:teuthology.orchestra.run.smithi052.stdout:Total 381 kB/s | 81 kB 00:00 2023-12-16T20:43:37.944 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:43:37.951 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:43:37.951 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:43:38.009 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:43:38.010 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:43:38.150 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:43:38.252 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:38.682 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:38.997 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:38.998 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:38.998 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:43:38.998 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:43:38.998 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:38.998 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:39.257 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-mirror 2023-12-16T20:43:39.838 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:37 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:39.989 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:39.990 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:39.990 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-12-16T20:43:39.991 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:39.991 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:43:39.991 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 2.9 M 2023-12-16T20:43:39.991 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:39.991 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:43:39.991 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:39.991 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-12-16T20:43:39.992 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:39.992 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 2.9 M 2023-12-16T20:43:39.992 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 12 M 2023-12-16T20:43:39.992 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:43:40.362 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-mirror-17.2.7-125.g1f2f5510.el9.x86_64.rpm 7.9 MB/s | 2.9 MB 00:00 2023-12-16T20:43:40.362 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:40.363 INFO:teuthology.orchestra.run.smithi052.stdout:Total 7.9 MB/s | 2.9 MB 00:00 2023-12-16T20:43:40.363 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:43:40.373 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:43:40.373 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:43:40.448 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:43:40.449 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:43:40.875 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:43:40.946 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:40.971 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:40.972 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T20:43:40.972 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-16T20:43:40.972 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T20:43:40.972 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T20:43:40.972 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:42.221 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:42.222 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:42.222 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:43:42.222 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:43:42.222 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:42.222 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:42.495 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-nbd 2023-12-16T20:43:43.086 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:41 ago on Sat 16 Dec 2023 08:42:02 PM UTC. 2023-12-16T20:43:43.239 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-12-16T20:43:43.240 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:43.240 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2023-12-16T20:43:43.240 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:43.240 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-12-16T20:43:43.240 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 162 k 2023-12-16T20:43:43.241 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:43.241 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-12-16T20:43:43.241 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-12-16T20:43:43.241 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-12-16T20:43:43.241 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:43.241 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 162 k 2023-12-16T20:43:43.241 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 465 k 2023-12-16T20:43:43.241 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-12-16T20:43:43.590 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-nbd-17.2.7-125.g1f2f5510.el9.x86_64.rpm 464 kB/s | 162 kB 00:00 2023-12-16T20:43:43.591 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:43:43.591 INFO:teuthology.orchestra.run.smithi052.stdout:Total 462 kB/s | 162 kB 00:00 2023-12-16T20:43:43.591 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-12-16T20:43:43.599 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-12-16T20:43:43.599 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-12-16T20:43:43.658 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-12-16T20:43:43.659 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-12-16T20:43:43.823 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-12-16T20:43:43.940 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:44.387 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:44.730 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T20:43:44.730 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:44.730 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-12-16T20:43:44.730 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T20:43:44.730 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:43:44.730 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-12-16T20:43:44.970 DEBUG:teuthology.parallel:result is None 2023-12-16T20:43:44.970 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T20:43:44.970 INFO:teuthology.packaging:ref: None 2023-12-16T20:43:44.970 INFO:teuthology.packaging:tag: None 2023-12-16T20:43:44.970 INFO:teuthology.packaging:branch: quincy 2023-12-16T20:43:44.970 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T20:43:44.971 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-16T20:43:45.264 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-16T20:43:45.286 INFO:teuthology.orchestra.run.smithi052.stdout:17.2.7-125.g1f2f5510.el9 2023-12-16T20:43:45.286 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-125.g1f2f5510.el9 2023-12-16T20:43:45.286 INFO:teuthology.task.install:The correct ceph version 17.2.7-125.g1f2f5510 is installed. 2023-12-16T20:43:45.288 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T20:43:45.288 INFO:teuthology.packaging:ref: None 2023-12-16T20:43:45.288 INFO:teuthology.packaging:tag: None 2023-12-16T20:43:45.288 INFO:teuthology.packaging:branch: quincy 2023-12-16T20:43:45.288 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T20:43:45.289 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-16T20:43:45.587 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-16T20:43:45.612 INFO:teuthology.orchestra.run.smithi150.stdout:17.2.7-125.g1f2f5510.el9 2023-12-16T20:43:45.612 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-125.g1f2f5510.el9 2023-12-16T20:43:45.613 INFO:teuthology.task.install:The correct ceph version 17.2.7-125.g1f2f5510 is installed. 2023-12-16T20:43:45.614 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-16T20:43:45.615 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:43:45.615 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-16T20:43:45.647 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-16T20:43:45.647 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-16T20:43:45.684 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-16T20:43:45.685 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:43:45.685 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/daemon-helper 2023-12-16T20:43:45.717 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-16T20:43:45.787 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-16T20:43:45.787 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2023-12-16T20:43:45.817 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-16T20:43:45.887 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-16T20:43:45.887 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:43:45.887 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-16T20:43:45.917 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-16T20:43:45.987 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-16T20:43:45.987 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-16T20:43:46.016 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-16T20:43:46.082 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-16T20:43:46.082 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:43:46.083 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/stdin-killer 2023-12-16T20:43:46.113 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-16T20:43:46.185 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-16T20:43:46.186 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2023-12-16T20:43:46.212 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-16T20:43:46.281 INFO:teuthology.run_tasks:Running task print... 2023-12-16T20:43:46.290 INFO:teuthology.task.print:**** done install task... 2023-12-16T20:43:46.291 INFO:teuthology.run_tasks:Running task cephadm... 2023-12-16T20:43:46.443 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'} 2023-12-16T20:43:46.444 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2023-12-16T20:43:46.444 INFO:tasks.cephadm:Cluster fsid is cf043a8a-9c53-11ee-95a5-87774f69a715 2023-12-16T20:43:46.444 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-12-16T20:43:46.444 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-12-16T20:43:46.444 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi052': '172.21.15.52', 'mon.smithi150': '172.21.15.150'} 2023-12-16T20:43:46.444 INFO:tasks.cephadm:Normalizing hostnames... 2023-12-16T20:43:46.445 DEBUG:teuthology.orchestra.run.smithi052:> sudo hostname $(hostname -s) 2023-12-16T20:43:46.477 DEBUG:teuthology.orchestra.run.smithi150:> sudo hostname $(hostname -s) 2023-12-16T20:43:46.505 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2023-12-16T20:43:46.506 DEBUG:teuthology.orchestra.run.smithi052:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-16T20:43:46.755 INFO:teuthology.orchestra.run.smithi052.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Dec 16 20:43 /home/ubuntu/cephtest/cephadm 2023-12-16T20:43:46.755 DEBUG:teuthology.orchestra.run.smithi150:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-16T20:43:46.943 INFO:teuthology.orchestra.run.smithi150.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Dec 16 20:43 /home/ubuntu/cephtest/cephadm 2023-12-16T20:43:46.943 DEBUG:teuthology.orchestra.run.smithi052:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-16T20:43:46.960 DEBUG:teuthology.orchestra.run.smithi150:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-16T20:43:46.984 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2023-12-16T20:43:46.985 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2023-12-16T20:43:47.003 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2023-12-16T20:43:47.733 INFO:teuthology.orchestra.run.smithi052.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-16T20:43:47.804 INFO:teuthology.orchestra.run.smithi150.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-16T20:44:29.928 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2023-12-16T20:44:29.928 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)", 2023-12-16T20:44:29.928 INFO:teuthology.orchestra.run.smithi150.stdout: "image_id": "60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5", 2023-12-16T20:44:29.928 INFO:teuthology.orchestra.run.smithi150.stdout: "repo_digests": [ 2023-12-16T20:44:29.928 INFO:teuthology.orchestra.run.smithi150.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112" 2023-12-16T20:44:29.928 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2023-12-16T20:44:29.928 INFO:teuthology.orchestra.run.smithi150.stdout:} 2023-12-16T20:44:32.640 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2023-12-16T20:44:32.641 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_version": "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)", 2023-12-16T20:44:32.641 INFO:teuthology.orchestra.run.smithi052.stdout: "image_id": "60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5", 2023-12-16T20:44:32.641 INFO:teuthology.orchestra.run.smithi052.stdout: "repo_digests": [ 2023-12-16T20:44:32.641 INFO:teuthology.orchestra.run.smithi052.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112" 2023-12-16T20:44:32.641 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2023-12-16T20:44:32.641 INFO:teuthology.orchestra.run.smithi052.stdout:} 2023-12-16T20:44:32.658 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph 2023-12-16T20:44:32.691 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph 2023-12-16T20:44:32.726 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /etc/ceph 2023-12-16T20:44:32.761 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /etc/ceph 2023-12-16T20:44:32.798 INFO:tasks.cephadm:Writing seed config... 2023-12-16T20:44:32.799 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2023-12-16T20:44:32.799 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2023-12-16T20:44:32.799 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-12-16T20:44:32.799 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-12-16T20:44:32.799 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-12-16T20:44:32.799 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-12-16T20:44:32.799 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-12-16T20:44:32.800 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-12-16T20:44:32.800 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-12-16T20:44:32.800 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-12-16T20:44:32.800 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-12-16T20:44:32.800 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-12-16T20:44:32.800 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-12-16T20:44:32.800 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-12-16T20:44:32.800 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-12-16T20:44:32.800 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-12-16T20:44:32.800 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-12-16T20:44:32.800 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-12-16T20:44:32.800 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-12-16T20:44:32.800 INFO:tasks.cephadm: override: [client] debug client = 20 2023-12-16T20:44:32.800 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-12-16T20:44:32.800 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-12-16T20:44:32.801 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-12-16T20:44:32.801 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-12-16T20:44:32.801 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-12-16T20:44:32.801 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-12-16T20:44:32.801 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-12-16T20:44:32.801 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-12-16T20:44:32.801 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-12-16T20:44:32.801 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-12-16T20:44:32.801 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-12-16T20:44:32.801 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-12-16T20:44:32.801 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-12-16T20:44:32.801 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-12-16T20:44:32.801 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-12-16T20:44:32.801 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-12-16T20:44:32.801 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-12-16T20:44:32.802 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-12-16T20:44:32.802 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-12-16T20:44:32.802 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-12-16T20:44:32.802 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:44:32.802 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-12-16T20:44:32.818 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = cf043a8a-9c53-11ee-95a5-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-12-16T20:44:32.818 DEBUG:teuthology.orchestra.run.smithi052:mon.smithi052> sudo journalctl -f -n 0 -u ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@mon.smithi052.service 2023-12-16T20:44:32.860 INFO:tasks.cephadm:Bootstrapping... 2023-12-16T20:44:32.861 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.52 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-12-16T20:44:33.075 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-12-16T20:44:33.075 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', 'cf043a8a-9c53-11ee-95a5-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.52', '--skip-admin-label'] 2023-12-16T20:44:33.101 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 4.7.2 2023-12-16T20:44:33.101 INFO:teuthology.orchestra.run.smithi052.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-12-16T20:44:33.102 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying podman|docker is present... 2023-12-16T20:44:33.129 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 4.7.2 2023-12-16T20:44:33.129 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying lvm2 is present... 2023-12-16T20:44:33.129 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying time synchronization is in place... 2023-12-16T20:44:33.135 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-12-16T20:44:33.135 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-12-16T20:44:33.141 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-12-16T20:44:33.141 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2023-12-16T20:44:33.148 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout enabled 2023-12-16T20:44:33.154 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout active 2023-12-16T20:44:33.155 INFO:teuthology.orchestra.run.smithi052.stdout:Unit chronyd.service is enabled and running 2023-12-16T20:44:33.155 INFO:teuthology.orchestra.run.smithi052.stdout:Repeating the final host check... 2023-12-16T20:44:33.179 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 4.7.2 2023-12-16T20:44:33.179 INFO:teuthology.orchestra.run.smithi052.stdout:podman (/bin/podman) version 4.7.2 is present 2023-12-16T20:44:33.179 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl is present 2023-12-16T20:44:33.179 INFO:teuthology.orchestra.run.smithi052.stdout:lvcreate is present 2023-12-16T20:44:33.186 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-12-16T20:44:33.186 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-12-16T20:44:33.192 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-12-16T20:44:33.193 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2023-12-16T20:44:33.199 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout enabled 2023-12-16T20:44:33.206 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout active 2023-12-16T20:44:33.206 INFO:teuthology.orchestra.run.smithi052.stdout:Unit chronyd.service is enabled and running 2023-12-16T20:44:33.206 INFO:teuthology.orchestra.run.smithi052.stdout:Host looks OK 2023-12-16T20:44:33.206 INFO:teuthology.orchestra.run.smithi052.stdout:Cluster fsid: cf043a8a-9c53-11ee-95a5-87774f69a715 2023-12-16T20:44:33.206 INFO:teuthology.orchestra.run.smithi052.stdout:Acquiring lock 140379691642944 on /run/cephadm/cf043a8a-9c53-11ee-95a5-87774f69a715.lock 2023-12-16T20:44:33.207 INFO:teuthology.orchestra.run.smithi052.stdout:Lock 140379691642944 acquired on /run/cephadm/cf043a8a-9c53-11ee-95a5-87774f69a715.lock 2023-12-16T20:44:33.207 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying IP 172.21.15.52 port 3300 ... 2023-12-16T20:44:33.207 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying IP 172.21.15.52 port 6789 ... 2023-12-16T20:44:33.207 INFO:teuthology.orchestra.run.smithi052.stdout:Base mon IP(s) is [172.21.15.52:3300, 172.21.15.52:6789], mon addrv is [v2:172.21.15.52:3300,v1:172.21.15.52:6789] 2023-12-16T20:44:33.211 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.52 metric 100 2023-12-16T20:44:33.211 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.52 metric 100 2023-12-16T20:44:33.214 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-12-16T20:44:33.214 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2023-12-16T20:44:33.214 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2023-12-16T20:44:33.217 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-12-16T20:44:33.217 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-12-16T20:44:33.217 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-12-16T20:44:33.217 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-12-16T20:44:33.217 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15de/64 scope link noprefixroute 2023-12-16T20:44:33.217 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-12-16T20:44:33.218 INFO:teuthology.orchestra.run.smithi052.stdout:Mon IP `172.21.15.52` is in CIDR network `172.21.0.0/20` 2023-12-16T20:44:33.218 INFO:teuthology.orchestra.run.smithi052.stdout:Mon IP `172.21.15.52` is in CIDR network `172.21.0.0/20` 2023-12-16T20:44:33.218 INFO:teuthology.orchestra.run.smithi052.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-12-16T20:44:33.218 INFO:teuthology.orchestra.run.smithi052.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-12-16T20:44:33.219 INFO:teuthology.orchestra.run.smithi052.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-16T20:44:33.707 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5 2023-12-16T20:44:33.707 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-16T20:44:33.707 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Getting image source signatures 2023-12-16T20:44:33.707 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying blob sha256:46af8f5390d4e94fc57efb422ccb97bb53dfe5b948546bfc191b46557eb2dbd9 2023-12-16T20:44:33.707 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying blob sha256:3c21fa6304a567e69d15522c48775b11468df75d53cc353263c55c753e1a9b4e 2023-12-16T20:44:33.708 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying config sha256:60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5 2023-12-16T20:44:33.708 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Writing manifest to image destination 2023-12-16T20:44:34.163 INFO:teuthology.orchestra.run.smithi052.stdout:ceph: stdout ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable) 2023-12-16T20:44:34.164 INFO:teuthology.orchestra.run.smithi052.stdout:Ceph version: ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable) 2023-12-16T20:44:34.164 INFO:teuthology.orchestra.run.smithi052.stdout:Extracting ceph user uid/gid from container image... 2023-12-16T20:44:34.487 INFO:teuthology.orchestra.run.smithi052.stdout:stat: stdout 167 167 2023-12-16T20:44:34.487 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial keys... 2023-12-16T20:44:34.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQAyDH5l/syTJRAAPJljtWASiNkkUS2gbCeOGg== 2023-12-16T20:44:35.188 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQAyDH5lgXjvORAAIqQq+q6GFpocnl4UYoj2mA== 2023-12-16T20:44:37.221 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQAzDH5lsoe6ExAAT0U/KDoWmimpY7KJQ8E67Q== 2023-12-16T20:44:37.222 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial monmap... 2023-12-16T20:44:40.086 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-16T20:44:40.086 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2023-12-16T20:44:40.087 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to cf043a8a-9c53-11ee-95a5-87774f69a715 2023-12-16T20:44:40.087 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-16T20:44:40.087 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool for smithi052 [v2:172.21.15.52:3300,v1:172.21.15.52:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-16T20:44:40.087 INFO:teuthology.orchestra.run.smithi052.stdout:setting min_mon_release = octopus 2023-12-16T20:44:40.087 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: set fsid to cf043a8a-9c53-11ee-95a5-87774f69a715 2023-12-16T20:44:40.087 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-16T20:44:40.087 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:44:40.087 INFO:teuthology.orchestra.run.smithi052.stdout:Creating mon... 2023-12-16T20:44:42.969 INFO:teuthology.orchestra.run.smithi052.stdout:create mon.smithi052 on 2023-12-16T20:44:43.215 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2023-12-16T20:44:43.475 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-12-16T20:44:43.678 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-cf043a8a-9c53-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-cf043a8a-9c53-11ee-95a5-87774f69a715.target. 2023-12-16T20:44:43.678 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-cf043a8a-9c53-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-cf043a8a-9c53-11ee-95a5-87774f69a715.target. 2023-12-16T20:44:43.995 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@mon.smithi052 2023-12-16T20:44:43.995 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to reset failed state of unit ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@mon.smithi052.service: Unit ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@mon.smithi052.service not loaded. 2023-12-16T20:44:44.326 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-cf043a8a-9c53-11ee-95a5-87774f69a715.target.wants/ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@mon.smithi052.service → /etc/systemd/system/ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@.service. 2023-12-16T20:44:44.591 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:44 smithi052 podman[80371]: 2023-12-16 20:44:44.45870112 +0000 UTC m=+0.018251386 image pull 60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5 quay.ceph.io/ceph-ci/ceph:quincy 2023-12-16T20:44:44.591 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:44 smithi052 podman[80371]: 2023-12-16 20:44:44.590105502 +0000 UTC m=+0.149655766 container init ffbb627d868101f93fddf77f15694b81085348cb3fe822f612beccbbe23bfcc1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052, org.label-schema.license=GPLv2, RELEASE=quincy-1f2f551, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.31.3, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2023-12-16T20:44:44.647 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-16T20:44:44.647 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2023-12-16T20:44:44.655 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-16T20:44:44.655 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2023-12-16T20:44:44.655 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2023-12-16T20:44:44.655 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to enable service . firewalld.service is not available 2023-12-16T20:44:44.655 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mon to start... 2023-12-16T20:44:44.656 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mon... 2023-12-16T20:44:44.848 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:44 smithi052 podman[80371]: 2023-12-16 20:44:44.606298468 +0000 UTC m=+0.165848720 container start ffbb627d868101f93fddf77f15694b81085348cb3fe822f612beccbbe23bfcc1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , RELEASE=quincy-1f2f551, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212) 2023-12-16T20:44:44.849 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:44 smithi052 bash[80371]: ffbb627d868101f93fddf77f15694b81085348cb3fe822f612beccbbe23bfcc1 2023-12-16T20:44:44.849 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:44 smithi052 systemd[1]: Started Ceph mon.smithi052 for cf043a8a-9c53-11ee-95a5-87774f69a715. 2023-12-16T20:44:44.849 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:44 smithi052 ceph-mon[80390]: mkfs cf043a8a-9c53-11ee-95a5-87774f69a715 2023-12-16T20:44:44.849 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:44 smithi052 ceph-mon[80390]: mon.smithi052 is new leader, mons smithi052 in quorum (ranks 0) 2023-12-16T20:44:45.259 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout cluster: 2023-12-16T20:44:45.259 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout id: cf043a8a-9c53-11ee-95a5-87774f69a715 2023-12-16T20:44:45.259 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-12-16T20:44:45.259 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2023-12-16T20:44:45.259 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout services: 2023-12-16T20:44:45.259 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi052 (age 0.360512s) 2023-12-16T20:44:45.259 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-12-16T20:44:45.259 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-12-16T20:44:45.259 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2023-12-16T20:44:45.259 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout data: 2023-12-16T20:44:45.260 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-12-16T20:44:45.260 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-12-16T20:44:45.260 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-12-16T20:44:45.260 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout pgs: 2023-12-16T20:44:45.260 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2023-12-16T20:44:45.260 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.945+0000 7f722d23a700 1 Processor -- start 2023-12-16T20:44:45.260 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.946+0000 7f722d23a700 1 -- start start 2023-12-16T20:44:45.260 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.946+0000 7f722d23a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7228076b80 0x7f7228104f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:45.260 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.946+0000 7f722d23a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72281054c0 con 0x7f7228076b80 2023-12-16T20:44:45.260 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.946+0000 7f7226d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7228076b80 0x7f7228104f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:45.260 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.946+0000 7f7226d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7228076b80 0x7f7228104f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43062/0 (socket says 172.21.15.52:43062) 2023-12-16T20:44:45.260 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.946+0000 7f7226d9d700 1 -- 172.21.15.52:0/3232956167 learned_addr learned my addr 172.21.15.52:0/3232956167 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:45.260 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.946+0000 7f7226d9d700 1 -- 172.21.15.52:0/3232956167 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7228105600 con 0x7f7228076b80 2023-12-16T20:44:45.260 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.947+0000 7f7226d9d700 1 --2- 172.21.15.52:0/3232956167 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7228076b80 0x7f7228104f80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f72200098e0 tx=0x7f7220009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7242dde542b4e5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:45.261 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.947+0000 7f7225d9b700 1 -- 172.21.15.52:0/3232956167 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f722000b860 con 0x7f7228076b80 2023-12-16T20:44:45.261 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.947+0000 7f7225d9b700 1 -- 172.21.15.52:0/3232956167 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722000b9c0 con 0x7f7228076b80 2023-12-16T20:44:45.261 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.947+0000 7f7225d9b700 1 -- 172.21.15.52:0/3232956167 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f722000bc90 con 0x7f7228076b80 2023-12-16T20:44:45.261 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.947+0000 7f722d23a700 1 -- 172.21.15.52:0/3232956167 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7228076b80 msgr2=0x7f7228104f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:45.261 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.947+0000 7f722d23a700 1 --2- 172.21.15.52:0/3232956167 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7228076b80 0x7f7228104f80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f72200098e0 tx=0x7f7220009250 comp rx=0 tx=0).stop 2023-12-16T20:44:45.261 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.947+0000 7f722d23a700 1 -- 172.21.15.52:0/3232956167 shutdown_connections 2023-12-16T20:44:45.261 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.947+0000 7f722d23a700 1 --2- 172.21.15.52:0/3232956167 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7228076b80 0x7f7228104f80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:45.261 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.947+0000 7f722d23a700 1 -- 172.21.15.52:0/3232956167 >> 172.21.15.52:0/3232956167 conn(0x7f72280fc950 msgr2=0x7f72280fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:45.261 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.947+0000 7f722d23a700 1 -- 172.21.15.52:0/3232956167 shutdown_connections 2023-12-16T20:44:45.261 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.948+0000 7f722d23a700 1 -- 172.21.15.52:0/3232956167 wait complete. 2023-12-16T20:44:45.261 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.948+0000 7f722d23a700 1 Processor -- start 2023-12-16T20:44:45.261 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.948+0000 7f722d23a700 1 -- start start 2023-12-16T20:44:45.261 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.948+0000 7f722d23a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7228076b80 0x7f72281109b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:45.261 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.948+0000 7f722d23a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7228110ef0 con 0x7f7228076b80 2023-12-16T20:44:45.262 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.949+0000 7f7226d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7228076b80 0x7f72281109b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:45.262 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.949+0000 7f7226d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7228076b80 0x7f72281109b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43072/0 (socket says 172.21.15.52:43072) 2023-12-16T20:44:45.262 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.949+0000 7f7226d9d700 1 -- 172.21.15.52:0/1449170756 learned_addr learned my addr 172.21.15.52:0/1449170756 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:45.262 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.949+0000 7f7226d9d700 1 -- 172.21.15.52:0/1449170756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72200094f0 con 0x7f7228076b80 2023-12-16T20:44:45.262 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.949+0000 7f7226d9d700 1 --2- 172.21.15.52:0/1449170756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7228076b80 0x7f72281109b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7220009290 tx=0x7f7220016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:45.262 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.949+0000 7f721f7fe700 1 -- 172.21.15.52:0/1449170756 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f722001c5c0 con 0x7f7228076b80 2023-12-16T20:44:45.262 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.949+0000 7f721f7fe700 1 -- 172.21.15.52:0/1449170756 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722001c740 con 0x7f7228076b80 2023-12-16T20:44:45.262 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.949+0000 7f722d23a700 1 -- 172.21.15.52:0/1449170756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72281110f0 con 0x7f7228076b80 2023-12-16T20:44:45.262 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.949+0000 7f721f7fe700 1 -- 172.21.15.52:0/1449170756 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f722001ca50 con 0x7f7228076b80 2023-12-16T20:44:45.262 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.949+0000 7f722d23a700 1 -- 172.21.15.52:0/1449170756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7228111500 con 0x7f7228076b80 2023-12-16T20:44:45.262 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.949+0000 7f721f7fe700 1 -- 172.21.15.52:0/1449170756 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f722001a450 con 0x7f7228076b80 2023-12-16T20:44:45.262 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.949+0000 7f721f7fe700 1 -- 172.21.15.52:0/1449170756 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f722002e790 con 0x7f7228076b80 2023-12-16T20:44:45.262 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.950+0000 7f722d23a700 1 -- 172.21.15.52:0/1449170756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f722804bb80 con 0x7f7228076b80 2023-12-16T20:44:45.262 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:44.952+0000 7f721f7fe700 1 -- 172.21.15.52:0/1449170756 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f722001ce30 con 0x7f7228076b80 2023-12-16T20:44:45.262 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.026+0000 7f722d23a700 1 -- 172.21.15.52:0/1449170756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f722810a0f0 con 0x7f7228076b80 2023-12-16T20:44:45.263 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.027+0000 7f721f7fe700 1 -- 172.21.15.52:0/1449170756 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f722001a840 con 0x7f7228076b80 2023-12-16T20:44:45.263 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.027+0000 7f722d23a700 1 -- 172.21.15.52:0/1449170756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7228076b80 msgr2=0x7f72281109b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:45.263 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.027+0000 7f722d23a700 1 --2- 172.21.15.52:0/1449170756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7228076b80 0x7f72281109b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7220009290 tx=0x7f7220016000 comp rx=0 tx=0).stop 2023-12-16T20:44:45.263 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.027+0000 7f722d23a700 1 -- 172.21.15.52:0/1449170756 shutdown_connections 2023-12-16T20:44:45.263 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.027+0000 7f722d23a700 1 --2- 172.21.15.52:0/1449170756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7228076b80 0x7f72281109b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:45.263 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.027+0000 7f722d23a700 1 -- 172.21.15.52:0/1449170756 >> 172.21.15.52:0/1449170756 conn(0x7f72280fc950 msgr2=0x7f72280fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:45.263 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.027+0000 7f722d23a700 1 -- 172.21.15.52:0/1449170756 shutdown_connections 2023-12-16T20:44:45.263 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.027+0000 7f722d23a700 1 -- 172.21.15.52:0/1449170756 wait complete. 2023-12-16T20:44:45.263 INFO:teuthology.orchestra.run.smithi052.stdout:mon is available 2023-12-16T20:44:45.263 INFO:teuthology.orchestra.run.smithi052.stdout:Assimilating anything we can from ceph.conf... 2023-12-16T20:44:45.840 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:45 smithi052 ceph-mon[80390]: mon.smithi052 is new leader, mons smithi052 in quorum (ranks 0) 2023-12-16T20:44:45.840 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:45 smithi052 ceph-mon[80390]: monmap e1: 1 mons at {smithi052=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T20:44:45.840 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:45 smithi052 ceph-mon[80390]: fsmap 2023-12-16T20:44:45.840 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:45 smithi052 ceph-mon[80390]: osdmap e1: 0 total, 0 up, 0 in 2023-12-16T20:44:45.840 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:45 smithi052 ceph-mon[80390]: mgrmap e1: no daemons active 2023-12-16T20:44:45.840 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:45 smithi052 ceph-mon[80390]: from='client.? 172.21.15.52:0/1449170756' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-12-16T20:44:45.840 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:45 smithi052 ceph-mon[80390]: from='client.? 172.21.15.52:0/4232469144' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-16T20:44:45.840 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:45 smithi052 ceph-mon[80390]: from='client.? 172.21.15.52:0/4232469144' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-16T20:44:45.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2023-12-16T20:44:45.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [global] 2023-12-16T20:44:45.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout fsid = cf043a8a-9c53-11ee-95a5-87774f69a715 2023-12-16T20:44:45.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.52:3300,v1:172.21.15.52:6789] 2023-12-16T20:44:45.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-12-16T20:44:45.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-12-16T20:44:45.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-12-16T20:44:45.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-12-16T20:44:45.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2023-12-16T20:44:45.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [mgr] 2023-12-16T20:44:45.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-12-16T20:44:45.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2023-12-16T20:44:45.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [osd] 2023-12-16T20:44:45.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-12-16T20:44:45.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-12-16T20:44:45.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-12-16T20:44:45.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-12-16T20:44:45.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-12-16T20:44:45.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.544+0000 7f89d5672700 1 Processor -- start 2023-12-16T20:44:45.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.544+0000 7f89d5672700 1 -- start start 2023-12-16T20:44:45.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.544+0000 7f89d5672700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89d01039c0 0x7f89d0103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:45.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.544+0000 7f89d5672700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d0104310 con 0x7f89d01039c0 2023-12-16T20:44:45.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.545+0000 7f89ceffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89d01039c0 0x7f89d0103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:45.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.545+0000 7f89ceffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89d01039c0 0x7f89d0103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43080/0 (socket says 172.21.15.52:43080) 2023-12-16T20:44:45.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.545+0000 7f89ceffd700 1 -- 172.21.15.52:0/1966881505 learned_addr learned my addr 172.21.15.52:0/1966881505 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:45.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.545+0000 7f89ceffd700 1 -- 172.21.15.52:0/1966881505 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d0101fb0 con 0x7f89d01039c0 2023-12-16T20:44:45.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.545+0000 7f89ceffd700 1 --2- 172.21.15.52:0/1966881505 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89d01039c0 0x7f89d0103dd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f89c8009e50 tx=0x7f89c8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a6e702ba7555aac server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:45.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.545+0000 7f89cdffb700 1 -- 172.21.15.52:0/1966881505 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89c80092e0 con 0x7f89d01039c0 2023-12-16T20:44:45.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.545+0000 7f89cdffb700 1 -- 172.21.15.52:0/1966881505 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c800ba20 con 0x7f89d01039c0 2023-12-16T20:44:45.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.545+0000 7f89cdffb700 1 -- 172.21.15.52:0/1966881505 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89c800bd30 con 0x7f89d01039c0 2023-12-16T20:44:45.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.546+0000 7f89d5672700 1 -- 172.21.15.52:0/1966881505 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89d01039c0 msgr2=0x7f89d0103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:45.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.546+0000 7f89d5672700 1 --2- 172.21.15.52:0/1966881505 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89d01039c0 0x7f89d0103dd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f89c8009e50 tx=0x7f89c8009840 comp rx=0 tx=0).stop 2023-12-16T20:44:45.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.546+0000 7f89d5672700 1 -- 172.21.15.52:0/1966881505 shutdown_connections 2023-12-16T20:44:45.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.546+0000 7f89d5672700 1 --2- 172.21.15.52:0/1966881505 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89d01039c0 0x7f89d0103dd0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:45.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.546+0000 7f89d5672700 1 -- 172.21.15.52:0/1966881505 >> 172.21.15.52:0/1966881505 conn(0x7f89d00fda40 msgr2=0x7f89d00ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:45.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.546+0000 7f89d5672700 1 -- 172.21.15.52:0/1966881505 shutdown_connections 2023-12-16T20:44:45.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.546+0000 7f89d5672700 1 -- 172.21.15.52:0/1966881505 wait complete. 2023-12-16T20:44:45.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.547+0000 7f89d5672700 1 Processor -- start 2023-12-16T20:44:45.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.547+0000 7f89d5672700 1 -- start start 2023-12-16T20:44:45.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.547+0000 7f89d5672700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89d01039c0 0x7f89d01109e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:45.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.547+0000 7f89d5672700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d0110f20 con 0x7f89d01039c0 2023-12-16T20:44:45.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.547+0000 7f89ceffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89d01039c0 0x7f89d01109e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:45.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.547+0000 7f89ceffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89d01039c0 0x7f89d01109e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43082/0 (socket says 172.21.15.52:43082) 2023-12-16T20:44:45.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.547+0000 7f89ceffd700 1 -- 172.21.15.52:0/4232469144 learned_addr learned my addr 172.21.15.52:0/4232469144 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:45.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.547+0000 7f89ceffd700 1 -- 172.21.15.52:0/4232469144 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c8009ae0 con 0x7f89d01039c0 2023-12-16T20:44:45.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.547+0000 7f89ceffd700 1 --2- 172.21.15.52:0/4232469144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89d01039c0 0x7f89d01109e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f89d0103020 tx=0x7f89c8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:45.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.548+0000 7f89c77fe700 1 -- 172.21.15.52:0/4232469144 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89c8012b90 con 0x7f89d01039c0 2023-12-16T20:44:45.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.548+0000 7f89c77fe700 1 -- 172.21.15.52:0/4232469144 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c8012d10 con 0x7f89d01039c0 2023-12-16T20:44:45.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.548+0000 7f89d5672700 1 -- 172.21.15.52:0/4232469144 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89d0111120 con 0x7f89d01039c0 2023-12-16T20:44:45.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.548+0000 7f89c77fe700 1 -- 172.21.15.52:0/4232469144 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89c801e400 con 0x7f89d01039c0 2023-12-16T20:44:45.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.548+0000 7f89d5672700 1 -- 172.21.15.52:0/4232469144 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89d0111dc0 con 0x7f89d01039c0 2023-12-16T20:44:45.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.548+0000 7f89c77fe700 1 -- 172.21.15.52:0/4232469144 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f89c8019030 con 0x7f89d01039c0 2023-12-16T20:44:45.856 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.548+0000 7f89c77fe700 1 -- 172.21.15.52:0/4232469144 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f89c801ed00 con 0x7f89d01039c0 2023-12-16T20:44:45.856 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.549+0000 7f89d5672700 1 -- 172.21.15.52:0/4232469144 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89d0111590 con 0x7f89d01039c0 2023-12-16T20:44:45.856 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.551+0000 7f89c77fe700 1 -- 172.21.15.52:0/4232469144 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f89c8017070 con 0x7f89d01039c0 2023-12-16T20:44:45.856 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.621+0000 7f89d5672700 1 -- 172.21.15.52:0/4232469144 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f89d010a1a0 con 0x7f89d01039c0 2023-12-16T20:44:45.856 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.624+0000 7f89c77fe700 1 -- 172.21.15.52:0/4232469144 <== mon.0 v2:172.21.15.52:3300/0 7 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f89c8039450 con 0x7f89d01039c0 2023-12-16T20:44:45.856 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.624+0000 7f89c77fe700 1 -- 172.21.15.52:0/4232469144 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f89c803b210 con 0x7f89d01039c0 2023-12-16T20:44:45.856 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.625+0000 7f89d5672700 1 -- 172.21.15.52:0/4232469144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89d01039c0 msgr2=0x7f89d01109e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:45.856 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.625+0000 7f89d5672700 1 --2- 172.21.15.52:0/4232469144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89d01039c0 0x7f89d01109e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f89d0103020 tx=0x7f89c8017000 comp rx=0 tx=0).stop 2023-12-16T20:44:45.856 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.625+0000 7f89d5672700 1 -- 172.21.15.52:0/4232469144 shutdown_connections 2023-12-16T20:44:45.856 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.625+0000 7f89d5672700 1 --2- 172.21.15.52:0/4232469144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89d01039c0 0x7f89d01109e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:45.856 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.625+0000 7f89d5672700 1 -- 172.21.15.52:0/4232469144 >> 172.21.15.52:0/4232469144 conn(0x7f89d00fda40 msgr2=0x7f89d0105040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:45.856 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.626+0000 7f89d5672700 1 -- 172.21.15.52:0/4232469144 shutdown_connections 2023-12-16T20:44:45.856 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:45.626+0000 7f89d5672700 1 -- 172.21.15.52:0/4232469144 wait complete. 2023-12-16T20:44:45.856 INFO:teuthology.orchestra.run.smithi052.stdout:Generating new minimal ceph.conf... 2023-12-16T20:44:46.449 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.127+0000 7fd11bba1700 1 Processor -- start 2023-12-16T20:44:46.449 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.127+0000 7fd11bba1700 1 -- start start 2023-12-16T20:44:46.449 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.128+0000 7fd11bba1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1141039c0 0x7fd114103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:46.450 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.128+0000 7fd11bba1700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd114104310 con 0x7fd1141039c0 2023-12-16T20:44:46.450 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.128+0000 7fd11993d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1141039c0 0x7fd114103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:46.450 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.128+0000 7fd11993d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1141039c0 0x7fd114103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43098/0 (socket says 172.21.15.52:43098) 2023-12-16T20:44:46.450 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.128+0000 7fd11993d700 1 -- 172.21.15.52:0/2904842929 learned_addr learned my addr 172.21.15.52:0/2904842929 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:46.450 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.128+0000 7fd11993d700 1 -- 172.21.15.52:0/2904842929 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd114101fb0 con 0x7fd1141039c0 2023-12-16T20:44:46.450 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.129+0000 7fd11993d700 1 --2- 172.21.15.52:0/2904842929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1141039c0 0x7fd114103dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd1100098e0 tx=0x7fd110009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e7b6ffc5a24e63 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:46.450 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.129+0000 7fd11893b700 1 -- 172.21.15.52:0/2904842929 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd11000b860 con 0x7fd1141039c0 2023-12-16T20:44:46.450 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.129+0000 7fd11893b700 1 -- 172.21.15.52:0/2904842929 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd11000be30 con 0x7fd1141039c0 2023-12-16T20:44:46.450 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.129+0000 7fd11893b700 1 -- 172.21.15.52:0/2904842929 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd110010380 con 0x7fd1141039c0 2023-12-16T20:44:46.450 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.129+0000 7fd11bba1700 1 -- 172.21.15.52:0/2904842929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1141039c0 msgr2=0x7fd114103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:46.450 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.129+0000 7fd11bba1700 1 --2- 172.21.15.52:0/2904842929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1141039c0 0x7fd114103dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd1100098e0 tx=0x7fd110009250 comp rx=0 tx=0).stop 2023-12-16T20:44:46.450 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.129+0000 7fd11bba1700 1 -- 172.21.15.52:0/2904842929 shutdown_connections 2023-12-16T20:44:46.450 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.129+0000 7fd11bba1700 1 --2- 172.21.15.52:0/2904842929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1141039c0 0x7fd114103dd0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:46.450 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.129+0000 7fd11bba1700 1 -- 172.21.15.52:0/2904842929 >> 172.21.15.52:0/2904842929 conn(0x7fd1140fda40 msgr2=0x7fd1140ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:46.451 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.129+0000 7fd11bba1700 1 -- 172.21.15.52:0/2904842929 shutdown_connections 2023-12-16T20:44:46.451 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.129+0000 7fd11bba1700 1 -- 172.21.15.52:0/2904842929 wait complete. 2023-12-16T20:44:46.454 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.130+0000 7fd11bba1700 1 Processor -- start 2023-12-16T20:44:46.454 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.130+0000 7fd11bba1700 1 -- start start 2023-12-16T20:44:46.454 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.130+0000 7fd11bba1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1141039c0 0x7fd114078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:46.454 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.130+0000 7fd11bba1700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd114077190 con 0x7fd1141039c0 2023-12-16T20:44:46.454 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.130+0000 7fd11993d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1141039c0 0x7fd114078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:46.455 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.130+0000 7fd11993d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1141039c0 0x7fd114078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43102/0 (socket says 172.21.15.52:43102) 2023-12-16T20:44:46.455 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.130+0000 7fd11993d700 1 -- 172.21.15.52:0/3586798824 learned_addr learned my addr 172.21.15.52:0/3586798824 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:46.455 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.131+0000 7fd11993d700 1 -- 172.21.15.52:0/3586798824 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1100094f0 con 0x7fd1141039c0 2023-12-16T20:44:46.455 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.131+0000 7fd11993d700 1 --2- 172.21.15.52:0/3586798824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1141039c0 0x7fd114078d10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd110006b00 tx=0x7fd110009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:46.455 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.131+0000 7fd10a7fc700 1 -- 172.21.15.52:0/3586798824 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1100109b0 con 0x7fd1141039c0 2023-12-16T20:44:46.455 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.131+0000 7fd10a7fc700 1 -- 172.21.15.52:0/3586798824 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd11001a430 con 0x7fd1141039c0 2023-12-16T20:44:46.456 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.131+0000 7fd11bba1700 1 -- 172.21.15.52:0/3586798824 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd114077390 con 0x7fd1141039c0 2023-12-16T20:44:46.456 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.131+0000 7fd10a7fc700 1 -- 172.21.15.52:0/3586798824 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1100245b0 con 0x7fd1141039c0 2023-12-16T20:44:46.456 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.131+0000 7fd11bba1700 1 -- 172.21.15.52:0/3586798824 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1140777a0 con 0x7fd1141039c0 2023-12-16T20:44:46.456 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.131+0000 7fd10a7fc700 1 -- 172.21.15.52:0/3586798824 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd110024a60 con 0x7fd1141039c0 2023-12-16T20:44:46.456 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.131+0000 7fd10a7fc700 1 -- 172.21.15.52:0/3586798824 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd11002eb80 con 0x7fd1141039c0 2023-12-16T20:44:46.456 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.132+0000 7fd11bba1700 1 -- 172.21.15.52:0/3586798824 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd11405f270 con 0x7fd1141039c0 2023-12-16T20:44:46.456 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.134+0000 7fd10a7fc700 1 -- 172.21.15.52:0/3586798824 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fd11002ed40 con 0x7fd1141039c0 2023-12-16T20:44:46.457 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.205+0000 7fd11bba1700 1 -- 172.21.15.52:0/3586798824 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd114102d60 con 0x7fd1141039c0 2023-12-16T20:44:46.457 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.205+0000 7fd10a7fc700 1 -- 172.21.15.52:0/3586798824 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fd11001a9c0 con 0x7fd1141039c0 2023-12-16T20:44:46.457 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.206+0000 7fd11bba1700 1 -- 172.21.15.52:0/3586798824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1141039c0 msgr2=0x7fd114078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:46.458 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.206+0000 7fd11bba1700 1 --2- 172.21.15.52:0/3586798824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1141039c0 0x7fd114078d10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd110006b00 tx=0x7fd110009250 comp rx=0 tx=0).stop 2023-12-16T20:44:46.458 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.206+0000 7fd11bba1700 1 -- 172.21.15.52:0/3586798824 shutdown_connections 2023-12-16T20:44:46.458 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.206+0000 7fd11bba1700 1 --2- 172.21.15.52:0/3586798824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1141039c0 0x7fd114078d10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:46.458 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.206+0000 7fd11bba1700 1 -- 172.21.15.52:0/3586798824 >> 172.21.15.52:0/3586798824 conn(0x7fd1140fda40 msgr2=0x7fd1140ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:46.458 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.206+0000 7fd11bba1700 1 -- 172.21.15.52:0/3586798824 shutdown_connections 2023-12-16T20:44:46.458 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:46.206+0000 7fd11bba1700 1 -- 172.21.15.52:0/3586798824 wait complete. 2023-12-16T20:44:46.459 INFO:teuthology.orchestra.run.smithi052.stdout:Restarting the monitor... 2023-12-16T20:44:46.694 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:46 smithi052 systemd[1]: Stopping Ceph mon.smithi052 for cf043a8a-9c53-11ee-95a5-87774f69a715... 2023-12-16T20:44:46.694 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80386]: 2023-12-16T20:44:46.641+0000 7f4eacef3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi052 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-12-16T20:44:46.694 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80386]: 2023-12-16T20:44:46.641+0000 7f4eacef3700 -1 mon.smithi052@0(leader) e1 *** Got Signal Terminated *** 2023-12-16T20:44:47.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:46 smithi052 podman[80636]: 2023-12-16 20:44:46.693098854 +0000 UTC m=+0.097199889 container died ffbb627d868101f93fddf77f15694b81085348cb3fe822f612beccbbe23bfcc1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052, io.buildah.version=1.31.3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-1f2f551, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2023-12-16T20:44:47.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:46 smithi052 podman[80636]: 2023-12-16 20:44:46.797743207 +0000 UTC m=+0.201844218 container cleanup ffbb627d868101f93fddf77f15694b81085348cb3fe822f612beccbbe23bfcc1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20231212, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.schema-version=1.0, RELEASE=quincy-1f2f551, maintainer=Guillaume Abrioux ) 2023-12-16T20:44:47.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:46 smithi052 bash[80636]: ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052 2023-12-16T20:44:47.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:46 smithi052 podman[80650]: 2023-12-16 20:44:46.880995064 +0000 UTC m=+0.183896900 container remove ffbb627d868101f93fddf77f15694b81085348cb3fe822f612beccbbe23bfcc1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052, GIT_CLEAN=True, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , RELEASE=quincy-1f2f551, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2023-12-16T20:44:47.372 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 systemd[1]: ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@mon.smithi052.service: Deactivated successfully. 2023-12-16T20:44:47.372 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 systemd[1]: Stopped Ceph mon.smithi052 for cf043a8a-9c53-11ee-95a5-87774f69a715. 2023-12-16T20:44:47.372 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 systemd[1]: Starting Ceph mon.smithi052 for cf043a8a-9c53-11ee-95a5-87774f69a715... 2023-12-16T20:44:47.372 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 podman[80741]: 2023-12-16 20:44:47.272576796 +0000 UTC m=+0.060400301 container create 094cffecdaf4e42894642076a08e43bf1d5979bb934b4a17e295705b21293ce1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052, RELEASE=quincy-1f2f551, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2023-12-16T20:44:47.372 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 podman[80741]: 2023-12-16 20:44:47.229831786 +0000 UTC m=+0.017655288 image pull 60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5 quay.ceph.io/ceph-ci/ceph:quincy 2023-12-16T20:44:47.372 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 podman[80741]: 2023-12-16 20:44:47.347974493 +0000 UTC m=+0.135797996 container init 094cffecdaf4e42894642076a08e43bf1d5979bb934b4a17e295705b21293ce1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.31.3, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-1f2f551, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2023-12-16T20:44:47.372 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 podman[80741]: 2023-12-16 20:44:47.35094168 +0000 UTC m=+0.138765181 container start 094cffecdaf4e42894642076a08e43bf1d5979bb934b4a17e295705b21293ce1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, CEPH_POINT_RELEASE=, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, RELEASE=quincy-1f2f551, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2023-12-16T20:44:47.373 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: set uid:gid to 167:167 (ceph:ceph) 2023-12-16T20:44:47.375 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable), process ceph-mon, pid 2 2023-12-16T20:44:47.376 INFO:teuthology.orchestra.run.smithi052.stdout:Setting mon public_network to 172.21.0.0/20 2023-12-16T20:44:47.635 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 bash[80741]: 094cffecdaf4e42894642076a08e43bf1d5979bb934b4a17e295705b21293ce1 2023-12-16T20:44:47.635 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: pidfile_write: ignore empty --pid-file 2023-12-16T20:44:47.635 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 systemd[1]: Started Ceph mon.smithi052 for cf043a8a-9c53-11ee-95a5-87774f69a715. 2023-12-16T20:44:47.636 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: load: jerasure load: lrc 2023-12-16T20:44:47.636 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: RocksDB version: 6.15.5 2023-12-16T20:44:47.636 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-16T20:44:47.636 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Compile date Dec 13 2023 2023-12-16T20:44:47.636 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: DB SUMMARY 2023-12-16T20:44:47.636 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: DB Session ID: 1TUCZTRSE3L5PBK8XJVM 2023-12-16T20:44:47.636 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: CURRENT file: CURRENT 2023-12-16T20:44:47.636 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: IDENTITY file: IDENTITY 2023-12-16T20:44:47.636 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2023-12-16T20:44:47.636 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi052/store.db dir, Total Num: 1, files: 000008.sst 2023-12-16T20:44:47.636 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi052/store.db: 000010.log size: 95072 ; 2023-12-16T20:44:47.636 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.error_if_exists: 0 2023-12-16T20:44:47.636 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.create_if_missing: 0 2023-12-16T20:44:47.636 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.paranoid_checks: 1 2023-12-16T20:44:47.636 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-12-16T20:44:47.637 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.env: 0x55d6ae20aa80 2023-12-16T20:44:47.637 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.fs: Posix File System 2023-12-16T20:44:47.637 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.info_log: 0x55d6af763000 2023-12-16T20:44:47.637 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_file_opening_threads: 16 2023-12-16T20:44:47.637 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.statistics: (nil) 2023-12-16T20:44:47.637 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.use_fsync: 0 2023-12-16T20:44:47.637 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_log_file_size: 0 2023-12-16T20:44:47.637 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-16T20:44:47.637 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.log_file_time_to_roll: 0 2023-12-16T20:44:47.637 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.keep_log_file_num: 1000 2023-12-16T20:44:47.637 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.recycle_log_file_num: 0 2023-12-16T20:44:47.637 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.allow_fallocate: 1 2023-12-16T20:44:47.637 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.allow_mmap_reads: 0 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.allow_mmap_writes: 0 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.use_direct_reads: 0 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.create_missing_column_families: 0 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.db_log_dir: 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi052/store.db 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.table_cache_numshardbits: 6 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.WAL_ttl_seconds: 0 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.WAL_size_limit_MB: 0 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.is_fd_close_on_exec: 1 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.advise_random_on_open: 1 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.db_write_buffer_size: 0 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.write_buffer_manager: 0x55d6b04d89f0 2023-12-16T20:44:47.638 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-16T20:44:47.639 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-16T20:44:47.639 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-16T20:44:47.639 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.use_adaptive_mutex: 0 2023-12-16T20:44:47.639 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.rate_limiter: (nil) 2023-12-16T20:44:47.639 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-16T20:44:47.639 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.wal_recovery_mode: 2 2023-12-16T20:44:47.639 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.enable_thread_tracking: 0 2023-12-16T20:44:47.639 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.enable_pipelined_write: 0 2023-12-16T20:44:47.639 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.unordered_write: 0 2023-12-16T20:44:47.639 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-16T20:44:47.639 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-16T20:44:47.639 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-16T20:44:47.639 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-16T20:44:47.639 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.row_cache: None 2023-12-16T20:44:47.639 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.wal_filter: None 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.allow_ingest_behind: 0 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.preserve_deletes: 0 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.two_write_queues: 0 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.manual_wal_flush: 0 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.atomic_flush: 0 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.persist_stats_to_disk: 0 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.write_dbid_to_manifest: 0 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.log_readahead_size: 0 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.best_efforts_recovery: 0 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.allow_data_in_errors: 0 2023-12-16T20:44:47.640 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.db_host_id: __hostname__ 2023-12-16T20:44:47.641 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_background_jobs: 2 2023-12-16T20:44:47.642 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_background_compactions: -1 2023-12-16T20:44:47.642 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_subcompactions: 1 2023-12-16T20:44:47.642 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-16T20:44:47.642 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-16T20:44:47.642 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.delayed_write_rate : 16777216 2023-12-16T20:44:47.642 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_total_wal_size: 0 2023-12-16T20:44:47.642 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-16T20:44:47.642 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.stats_dump_period_sec: 600 2023-12-16T20:44:47.642 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.stats_persist_period_sec: 600 2023-12-16T20:44:47.642 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-16T20:44:47.642 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_open_files: -1 2023-12-16T20:44:47.642 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.bytes_per_sync: 0 2023-12-16T20:44:47.642 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.wal_bytes_per_sync: 0 2023-12-16T20:44:47.643 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.strict_bytes_per_sync: 0 2023-12-16T20:44:47.643 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compaction_readahead_size: 0 2023-12-16T20:44:47.643 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_background_flushes: -1 2023-12-16T20:44:47.643 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Compression algorithms supported: 2023-12-16T20:44:47.643 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-16T20:44:47.643 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: kZSTD supported: 0 2023-12-16T20:44:47.643 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: kXpressCompression supported: 0 2023-12-16T20:44:47.643 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: kLZ4HCCompression supported: 1 2023-12-16T20:44:47.643 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: kZlibCompression supported: 1 2023-12-16T20:44:47.643 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: kSnappyCompression supported: 1 2023-12-16T20:44:47.643 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: kLZ4Compression supported: 1 2023-12-16T20:44:47.643 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: kBZip2Compression supported: 0 2023-12-16T20:44:47.643 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Fast CRC32 supported: Supported on x86 2023-12-16T20:44:47.643 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi052/store.db/MANIFEST-000009 2023-12-16T20:44:47.643 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.merge_operator: 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compaction_filter: None 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compaction_filter_factory: None 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.sst_partitioner_factory: None 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.memtable_factory: SkipListFactory 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.table_factory: BlockBasedTable 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d6af7b8f90) 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: cache_index_and_filter_blocks: 1 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: pin_top_level_index_and_filter: 1 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: index_type: 0 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: data_block_index_type: 0 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: index_shortening: 1 2023-12-16T20:44:47.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: data_block_hash_table_util_ratio: 0.750000 2023-12-16T20:44:47.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: hash_index_allow_collision: 1 2023-12-16T20:44:47.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: checksum: 1 2023-12-16T20:44:47.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: no_block_cache: 0 2023-12-16T20:44:47.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: block_cache: 0x55d6af7eb090 2023-12-16T20:44:47.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: block_cache_name: BinnedLRUCache 2023-12-16T20:44:47.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: block_cache_options: 2023-12-16T20:44:47.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: capacity : 536870912 2023-12-16T20:44:47.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: num_shard_bits : 4 2023-12-16T20:44:47.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: strict_capacity_limit : 0 2023-12-16T20:44:47.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: high_pri_pool_ratio: 0.000 2023-12-16T20:44:47.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: block_cache_compressed: (nil) 2023-12-16T20:44:47.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: persistent_cache: (nil) 2023-12-16T20:44:47.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: block_size: 4096 2023-12-16T20:44:47.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: block_size_deviation: 10 2023-12-16T20:44:47.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: block_restart_interval: 16 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: index_block_restart_interval: 1 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: metadata_block_size: 4096 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: partition_filters: 0 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: use_delta_encoding: 1 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: whole_key_filtering: 1 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: verify_compression: 0 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: read_amp_bytes_per_bit: 0 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: format_version: 4 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: enable_index_compression: 1 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: block_align: 0 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.write_buffer_size: 33554432 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_write_buffer_number: 2 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compression: NoCompression 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.bottommost_compression: Disabled 2023-12-16T20:44:47.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.prefix_extractor: nullptr 2023-12-16T20:44:47.647 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-16T20:44:47.647 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.num_levels: 7 2023-12-16T20:44:47.647 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-16T20:44:47.647 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-16T20:44:47.647 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-16T20:44:47.647 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-16T20:44:47.647 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-16T20:44:47.647 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-16T20:44:47.647 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-16T20:44:47.647 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-16T20:44:47.647 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-12-16T20:44:47.647 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-16T20:44:47.647 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compression_opts.window_bits: -14 2023-12-16T20:44:47.647 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compression_opts.level: 32767 2023-12-16T20:44:47.647 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compression_opts.strategy: 0 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compression_opts.enabled: false 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.target_file_size_base: 67108864 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.target_file_size_multiplier: 1 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-16T20:44:47.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-16T20:44:47.649 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-16T20:44:47.649 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-16T20:44:47.649 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-16T20:44:47.649 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-16T20:44:47.649 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-16T20:44:47.649 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.arena_block_size: 4194304 2023-12-16T20:44:47.649 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-16T20:44:47.649 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-16T20:44:47.649 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-16T20:44:47.649 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.disable_auto_compactions: 0 2023-12-16T20:44:47.649 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-16T20:44:47.649 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-16T20:44:47.649 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-16T20:44:47.649 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-16T20:44:47.649 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-16T20:44:47.650 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-16T20:44:47.650 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-16T20:44:47.650 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-16T20:44:47.650 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-16T20:44:47.650 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-16T20:44:47.650 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.table_properties_collectors: 2023-12-16T20:44:47.650 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.inplace_update_support: 0 2023-12-16T20:44:47.650 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.inplace_update_num_locks: 10000 2023-12-16T20:44:47.650 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-16T20:44:47.650 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-16T20:44:47.650 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.memtable_huge_page_size: 0 2023-12-16T20:44:47.650 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.bloom_locality: 0 2023-12-16T20:44:47.650 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.max_successive_merges: 0 2023-12-16T20:44:47.650 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.optimize_filters_for_hits: 0 2023-12-16T20:44:47.650 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.paranoid_file_checks: 0 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.force_consistency_checks: 1 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.report_bg_io_stats: 0 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.ttl: 2592000 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.periodic_compaction_seconds: 0 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.enable_blob_files: false 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.min_blob_size: 0 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.blob_file_size: 268435456 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.blob_compression_type: NoCompression 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.enable_blob_garbage_collection: false 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi052/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: EVENT_LOG_v1 {"time_micros": 1702759487379198, "job": 1, "event": "recovery_started", "wal_files": [10]} 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2023-12-16T20:44:47.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-12-16T20:44:47.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: EVENT_LOG_v1 {"time_micros": 1702759487380023, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 90708, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 88821, "index_size": 247, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13140, "raw_average_key_size": 50, "raw_value_size": 81644, "raw_average_value_size": 311, "num_data_blocks": 10, "num_entries": 262, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1702759487, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "6be92b34-f4ee-4a8c-8650-567f26563d1b", "db_session_id": "1TUCZTRSE3L5PBK8XJVM"}} 2023-12-16T20:44:47.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2023-12-16T20:44:47.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: EVENT_LOG_v1 {"time_micros": 1702759487380529, "job": 1, "event": "recovery_finished"} 2023-12-16T20:44:47.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi052/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-12-16T20:44:47.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55d6af77f180 2023-12-16T20:44:47.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: DB pointer 0x55d6af854000 2023-12-16T20:44:47.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2023-12-16T20:44:47.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: rocksdb: [db/db_impl/db_impl.cc:903] 2023-12-16T20:44:47.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: ** DB Stats ** 2023-12-16T20:44:47.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T20:44:47.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-16T20:44:47.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-16T20:44:47.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T20:44:47.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-16T20:44:47.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-16T20:44:47.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T20:44:47.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: 2023-12-16T20:44:47.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: ** Compaction Stats [default] ** 2023-12-16T20:44:47.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T20:44:47.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T20:44:47.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: L0 2/0 90.38 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.6 0.00 0.00 1 0.001 0 0 2023-12-16T20:44:47.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Sum 2/0 90.38 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.6 0.00 0.00 1 0.001 0 0 2023-12-16T20:44:47.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.6 0.00 0.00 1 0.001 0 0 2023-12-16T20:44:47.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: 2023-12-16T20:44:47.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: ** Compaction Stats [default] ** 2023-12-16T20:44:47.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T20:44:47.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T20:44:47.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 159.6 0.00 0.00 1 0.001 0 0 2023-12-16T20:44:47.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T20:44:47.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-12-16T20:44:47.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T20:44:47.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T20:44:47.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T20:44:47.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: AddFile(Keys): cumulative 0, interval 0 2023-12-16T20:44:47.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Cumulative compaction: 0.00 GB write, 23.06 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T20:44:47.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Interval compaction: 0.00 GB write, 23.06 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T20:44:47.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T20:44:47.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: 2023-12-16T20:44:47.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: ** File Read Latency Histogram By Level [default] ** 2023-12-16T20:44:47.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: 2023-12-16T20:44:47.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: ** Compaction Stats [default] ** 2023-12-16T20:44:47.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T20:44:47.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T20:44:47.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: L0 2/0 90.38 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.6 0.00 0.00 1 0.001 0 0 2023-12-16T20:44:47.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Sum 2/0 90.38 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.6 0.00 0.00 1 0.001 0 0 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: ** Compaction Stats [default] ** 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 159.6 0.00 0.00 1 0.001 0 0 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: AddFile(Keys): cumulative 0, interval 0 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Cumulative compaction: 0.00 GB write, 22.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T20:44:47.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: 2023-12-16T20:44:47.656 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: ** File Read Latency Histogram By Level [default] ** 2023-12-16T20:44:47.656 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: starting mon.smithi052 rank 0 at public addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] at bind addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi052 fsid cf043a8a-9c53-11ee-95a5-87774f69a715 2023-12-16T20:44:47.656 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: mon.smithi052@-1(???) e1 preinit fsid cf043a8a-9c53-11ee-95a5-87774f69a715 2023-12-16T20:44:47.656 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: mon.smithi052@-1(???).mds e1 new map 2023-12-16T20:44:47.656 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: mon.smithi052@-1(???).mds e1 print_map 2023-12-16T20:44:47.656 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: e1 2023-12-16T20:44:47.656 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-12-16T20:44:47.656 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T20:44:47.656 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: legacy client fscid: -1 2023-12-16T20:44:47.656 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: 2023-12-16T20:44:47.656 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout: No filesystems configured 2023-12-16T20:44:47.656 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: mon.smithi052@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-16T20:44:47.656 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: mon.smithi052@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T20:44:47.656 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: mon.smithi052@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T20:44:47.656 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: mon.smithi052@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T20:44:47.657 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: mon.smithi052@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-12-16T20:44:47.657 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: mon.smithi052 is new leader, mons smithi052 in quorum (ranks 0) 2023-12-16T20:44:47.657 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: monmap e1: 1 mons at {smithi052=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T20:44:47.657 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: fsmap 2023-12-16T20:44:47.657 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: osdmap e1: 0 total, 0 up, 0 in 2023-12-16T20:44:47.657 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:47 smithi052 ceph-mon[80761]: mgrmap e1: no daemons active 2023-12-16T20:44:47.947 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.632+0000 7f1aaac6c700 1 Processor -- start 2023-12-16T20:44:47.948 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.632+0000 7f1aaac6c700 1 -- start start 2023-12-16T20:44:47.948 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.632+0000 7f1aaac6c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa4078ce0 0x7f1aa40771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:47.948 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.632+0000 7f1aaac6c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa40790f0 con 0x7f1aa4078ce0 2023-12-16T20:44:47.948 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.633+0000 7f1aa8a08700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa4078ce0 0x7f1aa40771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:47.948 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.633+0000 7f1aa8a08700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa4078ce0 0x7f1aa40771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43114/0 (socket says 172.21.15.52:43114) 2023-12-16T20:44:47.948 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.633+0000 7f1aa8a08700 1 -- 172.21.15.52:0/2526950853 learned_addr learned my addr 172.21.15.52:0/2526950853 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:47.948 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.633+0000 7f1aa8a08700 1 -- 172.21.15.52:0/2526950853 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa4077730 con 0x7f1aa4078ce0 2023-12-16T20:44:47.948 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.633+0000 7f1aa8a08700 1 --2- 172.21.15.52:0/2526950853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa4078ce0 0x7f1aa40771f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009e50 tx=0x7f1a9c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=908eeb656bc75baa server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:47.948 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.633+0000 7f1aa37fe700 1 -- 172.21.15.52:0/2526950853 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9c0092e0 con 0x7f1aa4078ce0 2023-12-16T20:44:47.948 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.634+0000 7f1aa37fe700 1 -- 172.21.15.52:0/2526950853 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1a9c00be90 con 0x7f1aa4078ce0 2023-12-16T20:44:47.948 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.634+0000 7f1aa37fe700 1 -- 172.21.15.52:0/2526950853 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9c007400 con 0x7f1aa4078ce0 2023-12-16T20:44:47.949 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.634+0000 7f1aaac6c700 1 -- 172.21.15.52:0/2526950853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa4078ce0 msgr2=0x7f1aa40771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:47.949 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.634+0000 7f1aaac6c700 1 --2- 172.21.15.52:0/2526950853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa4078ce0 0x7f1aa40771f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009e50 tx=0x7f1a9c009840 comp rx=0 tx=0).stop 2023-12-16T20:44:47.949 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.634+0000 7f1aaac6c700 1 -- 172.21.15.52:0/2526950853 shutdown_connections 2023-12-16T20:44:47.949 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.635+0000 7f1aaac6c700 1 --2- 172.21.15.52:0/2526950853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa4078ce0 0x7f1aa40771f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:47.949 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.635+0000 7f1aaac6c700 1 -- 172.21.15.52:0/2526950853 >> 172.21.15.52:0/2526950853 conn(0x7f1aa40fda10 msgr2=0x7f1aa40ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:47.949 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.635+0000 7f1aaac6c700 1 -- 172.21.15.52:0/2526950853 shutdown_connections 2023-12-16T20:44:47.949 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.635+0000 7f1aaac6c700 1 -- 172.21.15.52:0/2526950853 wait complete. 2023-12-16T20:44:47.949 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.635+0000 7f1aaac6c700 1 Processor -- start 2023-12-16T20:44:47.949 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.635+0000 7f1aaac6c700 1 -- start start 2023-12-16T20:44:47.949 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.636+0000 7f1aaac6c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa4078ce0 0x7f1aa41133a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:47.950 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.636+0000 7f1aaac6c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa41138e0 con 0x7f1aa4078ce0 2023-12-16T20:44:47.950 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.636+0000 7f1aa8a08700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa4078ce0 0x7f1aa41133a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:47.950 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.636+0000 7f1aa8a08700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa4078ce0 0x7f1aa41133a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43122/0 (socket says 172.21.15.52:43122) 2023-12-16T20:44:47.950 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.636+0000 7f1aa8a08700 1 -- 172.21.15.52:0/397849447 learned_addr learned my addr 172.21.15.52:0/397849447 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:47.950 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.636+0000 7f1aa8a08700 1 -- 172.21.15.52:0/397849447 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c009ae0 con 0x7f1aa4078ce0 2023-12-16T20:44:47.950 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.636+0000 7f1aa8a08700 1 --2- 172.21.15.52:0/397849447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa4078ce0 0x7f1aa41133a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009ab0 tx=0x7f1a9c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:47.950 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.637+0000 7f1aa17fa700 1 -- 172.21.15.52:0/397849447 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9c007670 con 0x7f1aa4078ce0 2023-12-16T20:44:47.950 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.637+0000 7f1aa17fa700 1 -- 172.21.15.52:0/397849447 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1a9c007c40 con 0x7f1aa4078ce0 2023-12-16T20:44:47.950 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.637+0000 7f1aa17fa700 1 -- 172.21.15.52:0/397849447 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9c01cb10 con 0x7f1aa4078ce0 2023-12-16T20:44:47.950 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.637+0000 7f1aaac6c700 1 -- 172.21.15.52:0/397849447 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa4113ae0 con 0x7f1aa4078ce0 2023-12-16T20:44:47.950 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.637+0000 7f1aaac6c700 1 -- 172.21.15.52:0/397849447 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa4114780 con 0x7f1aa4078ce0 2023-12-16T20:44:47.950 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.637+0000 7f1aa17fa700 1 -- 172.21.15.52:0/397849447 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1a9c019070 con 0x7f1aa4078ce0 2023-12-16T20:44:47.950 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.637+0000 7f1aa17fa700 1 -- 172.21.15.52:0/397849447 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1a9c025d20 con 0x7f1aa4078ce0 2023-12-16T20:44:47.950 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.638+0000 7f1aaac6c700 1 -- 172.21.15.52:0/397849447 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa4113f50 con 0x7f1aa4078ce0 2023-12-16T20:44:47.951 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.641+0000 7f1aa17fa700 1 -- 172.21.15.52:0/397849447 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1a9c02a060 con 0x7f1aa4078ce0 2023-12-16T20:44:47.951 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.711+0000 7f1aaac6c700 1 -- 172.21.15.52:0/397849447 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f1aa405f270 con 0x7f1aa4078ce0 2023-12-16T20:44:47.951 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.713+0000 7f1aa17fa700 1 -- 172.21.15.52:0/397849447 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1a9c020070 con 0x7f1aa4078ce0 2023-12-16T20:44:47.951 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.714+0000 7f1aaac6c700 1 -- 172.21.15.52:0/397849447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa4078ce0 msgr2=0x7f1aa41133a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:47.951 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.714+0000 7f1aaac6c700 1 --2- 172.21.15.52:0/397849447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa4078ce0 0x7f1aa41133a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009ab0 tx=0x7f1a9c009840 comp rx=0 tx=0).stop 2023-12-16T20:44:47.951 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.714+0000 7f1aaac6c700 1 -- 172.21.15.52:0/397849447 shutdown_connections 2023-12-16T20:44:47.951 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.714+0000 7f1aaac6c700 1 --2- 172.21.15.52:0/397849447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa4078ce0 0x7f1aa41133a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:47.951 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.714+0000 7f1aaac6c700 1 -- 172.21.15.52:0/397849447 >> 172.21.15.52:0/397849447 conn(0x7f1aa40fda10 msgr2=0x7f1aa40ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:47.951 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.714+0000 7f1aaac6c700 1 -- 172.21.15.52:0/397849447 shutdown_connections 2023-12-16T20:44:47.951 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:47.714+0000 7f1aaac6c700 1 -- 172.21.15.52:0/397849447 wait complete. 2023-12-16T20:44:47.951 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote config to /etc/ceph/ceph.conf 2023-12-16T20:44:47.951 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-12-16T20:44:47.951 INFO:teuthology.orchestra.run.smithi052.stdout:Creating mgr... 2023-12-16T20:44:47.951 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying port 9283 ... 2023-12-16T20:44:48.126 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@mgr.smithi052.nmkcbs 2023-12-16T20:44:48.126 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to reset failed state of unit ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@mgr.smithi052.nmkcbs.service: Unit ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@mgr.smithi052.nmkcbs.service not loaded. 2023-12-16T20:44:48.321 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-cf043a8a-9c53-11ee-95a5-87774f69a715.target.wants/ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@mgr.smithi052.nmkcbs.service → /etc/systemd/system/ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@.service. 2023-12-16T20:44:48.687 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-16T20:44:48.687 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2023-12-16T20:44:48.694 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-16T20:44:48.694 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2023-12-16T20:44:48.694 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2023-12-16T20:44:48.694 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to enable service . firewalld.service is not available 2023-12-16T20:44:48.703 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-16T20:44:48.704 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2023-12-16T20:44:48.712 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-16T20:44:48.712 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2023-12-16T20:44:48.713 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2023-12-16T20:44:48.713 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2023-12-16T20:44:48.713 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr to start... 2023-12-16T20:44:48.713 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr... 2023-12-16T20:44:48.941 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:48 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/397849447' entity='client.admin' 2023-12-16T20:44:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:49 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/728192951' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-16T20:44:50.991 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2023-12-16T20:44:50.994 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2023-12-16T20:44:50.994 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "cf043a8a-9c53-11ee-95a5-87774f69a715", 2023-12-16T20:44:50.994 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2023-12-16T20:44:50.994 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-12-16T20:44:50.994 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-12-16T20:44:50.994 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-12-16T20:44:50.994 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:50.994 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-12-16T20:44:50.994 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-12-16T20:44:50.994 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2023-12-16T20:44:50.994 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2023-12-16T20:44:50.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-12-16T20:44:50.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "smithi052" 2023-12-16T20:44:50.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2023-12-16T20:44:50.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 1, 2023-12-16T20:44:50.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2023-12-16T20:44:50.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:50.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-12-16T20:44:50.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-12-16T20:44:50.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:50.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-12-16T20:44:50.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:50.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-12-16T20:44:50.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-12-16T20:44:50.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-12-16T20:44:51.002 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-12-16T20:44:51.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-12-16T20:44:51.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-12-16T20:44:51.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:51.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-12-16T20:44:51.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-12-16T20:44:51.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-12-16T20:44:51.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-12-16T20:44:51.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-12-16T20:44:51.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-12-16T20:44:51.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-12-16T20:44:51.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-12-16T20:44:51.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-12-16T20:44:51.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:51.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-12-16T20:44:51.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:51.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2023-12-16T20:44:44.680846+0000", 2023-12-16T20:44:51.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T20:44:51.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:51.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-12-16T20:44:51.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2023-12-16T20:44:51.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.081+0000 7fabd359e700 1 Processor -- start 2023-12-16T20:44:51.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.082+0000 7fabd359e700 1 -- start start 2023-12-16T20:44:51.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.082+0000 7fabd359e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabd4070f20 0x7fabd4104dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:51.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.082+0000 7fabd359e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd4105310 con 0x7fabd4070f20 2023-12-16T20:44:51.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.082+0000 7fabd259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabd4070f20 0x7fabd4104dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:51.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.082+0000 7fabd259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabd4070f20 0x7fabd4104dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43146/0 (socket says 172.21.15.52:43146) 2023-12-16T20:44:51.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.082+0000 7fabd259c700 1 -- 172.21.15.52:0/929315997 learned_addr learned my addr 172.21.15.52:0/929315997 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:51.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.083+0000 7fabd259c700 1 -- 172.21.15.52:0/929315997 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd4105450 con 0x7fabd4070f20 2023-12-16T20:44:51.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.083+0000 7fabd259c700 1 --2- 172.21.15.52:0/929315997 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabd4070f20 0x7fabd4104dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fabc40098e0 tx=0x7fabc4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7158397068e6e887 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:51.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.083+0000 7fabd159a700 1 -- 172.21.15.52:0/929315997 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabc400b860 con 0x7fabd4070f20 2023-12-16T20:44:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.083+0000 7fabd159a700 1 -- 172.21.15.52:0/929315997 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fabc400be30 con 0x7fabd4070f20 2023-12-16T20:44:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.083+0000 7fabd159a700 1 -- 172.21.15.52:0/929315997 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabc4010380 con 0x7fabd4070f20 2023-12-16T20:44:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.083+0000 7fabd359e700 1 -- 172.21.15.52:0/929315997 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabd4070f20 msgr2=0x7fabd4104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.083+0000 7fabd359e700 1 --2- 172.21.15.52:0/929315997 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabd4070f20 0x7fabd4104dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fabc40098e0 tx=0x7fabc4009250 comp rx=0 tx=0).stop 2023-12-16T20:44:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.084+0000 7fabd359e700 1 -- 172.21.15.52:0/929315997 shutdown_connections 2023-12-16T20:44:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.084+0000 7fabd359e700 1 --2- 172.21.15.52:0/929315997 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabd4070f20 0x7fabd4104dd0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.084+0000 7fabd359e700 1 -- 172.21.15.52:0/929315997 >> 172.21.15.52:0/929315997 conn(0x7fabd40fc760 msgr2=0x7fabd40febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.084+0000 7fabd359e700 1 -- 172.21.15.52:0/929315997 shutdown_connections 2023-12-16T20:44:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.084+0000 7fabd359e700 1 -- 172.21.15.52:0/929315997 wait complete. 2023-12-16T20:44:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.084+0000 7fabd359e700 1 Processor -- start 2023-12-16T20:44:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.084+0000 7fabd359e700 1 -- start start 2023-12-16T20:44:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.084+0000 7fabd359e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabd4070f20 0x7fabd4110f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.084+0000 7fabd359e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd41114c0 con 0x7fabd4070f20 2023-12-16T20:44:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.085+0000 7fabd259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabd4070f20 0x7fabd4110f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.085+0000 7fabd259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabd4070f20 0x7fabd4110f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43152/0 (socket says 172.21.15.52:43152) 2023-12-16T20:44:51.007 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.085+0000 7fabd259c700 1 -- 172.21.15.52:0/728192951 learned_addr learned my addr 172.21.15.52:0/728192951 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:51.007 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.085+0000 7fabd259c700 1 -- 172.21.15.52:0/728192951 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc40094f0 con 0x7fabd4070f20 2023-12-16T20:44:51.007 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.085+0000 7fabd259c700 1 --2- 172.21.15.52:0/728192951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabd4070f20 0x7fabd4110f80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fabc4006b00 tx=0x7fabc4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:51.007 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.085+0000 7fabcaffd700 1 -- 172.21.15.52:0/728192951 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabc4010a80 con 0x7fabd4070f20 2023-12-16T20:44:51.007 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.085+0000 7fabcaffd700 1 -- 172.21.15.52:0/728192951 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fabc401a430 con 0x7fabd4070f20 2023-12-16T20:44:51.007 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.085+0000 7fabd359e700 1 -- 172.21.15.52:0/728192951 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd41116c0 con 0x7fabd4070f20 2023-12-16T20:44:51.007 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.085+0000 7fabd359e700 1 -- 172.21.15.52:0/728192951 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd4111ad0 con 0x7fabd4070f20 2023-12-16T20:44:51.007 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.085+0000 7fabcaffd700 1 -- 172.21.15.52:0/728192951 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabc40245b0 con 0x7fabd4070f20 2023-12-16T20:44:51.007 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.085+0000 7fabcaffd700 1 -- 172.21.15.52:0/728192951 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fabc4024a60 con 0x7fabd4070f20 2023-12-16T20:44:51.007 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.086+0000 7fabcaffd700 1 -- 172.21.15.52:0/728192951 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fabc402ec40 con 0x7fabd4070f20 2023-12-16T20:44:51.007 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.086+0000 7fabd359e700 1 -- 172.21.15.52:0/728192951 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd404bb80 con 0x7fabd4070f20 2023-12-16T20:44:51.007 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.088+0000 7fabcaffd700 1 -- 172.21.15.52:0/728192951 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fabc402e440 con 0x7fabd4070f20 2023-12-16T20:44:51.007 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.163+0000 7fabd359e700 1 -- 172.21.15.52:0/728192951 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fabd4106170 con 0x7fabd4070f20 2023-12-16T20:44:51.007 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.164+0000 7fabcaffd700 1 -- 172.21.15.52:0/728192951 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fabc4010be0 con 0x7fabd4070f20 2023-12-16T20:44:51.007 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.165+0000 7fabd359e700 1 -- 172.21.15.52:0/728192951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabd4070f20 msgr2=0x7fabd4110f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:51.008 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.165+0000 7fabd359e700 1 --2- 172.21.15.52:0/728192951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabd4070f20 0x7fabd4110f80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fabc4006b00 tx=0x7fabc4009250 comp rx=0 tx=0).stop 2023-12-16T20:44:51.008 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.165+0000 7fabd359e700 1 -- 172.21.15.52:0/728192951 shutdown_connections 2023-12-16T20:44:51.008 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.165+0000 7fabd359e700 1 --2- 172.21.15.52:0/728192951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabd4070f20 0x7fabd4110f80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:51.008 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.165+0000 7fabd359e700 1 -- 172.21.15.52:0/728192951 >> 172.21.15.52:0/728192951 conn(0x7fabd40fc760 msgr2=0x7fabd40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:51.008 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.165+0000 7fabd359e700 1 -- 172.21.15.52:0/728192951 shutdown_connections 2023-12-16T20:44:51.008 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:49.165+0000 7fabd359e700 1 -- 172.21.15.52:0/728192951 wait complete. 2023-12-16T20:44:51.008 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (1/15)... 2023-12-16T20:44:53.561 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2023-12-16T20:44:53.561 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2023-12-16T20:44:53.561 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "cf043a8a-9c53-11ee-95a5-87774f69a715", 2023-12-16T20:44:53.562 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2023-12-16T20:44:53.562 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-12-16T20:44:53.562 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-12-16T20:44:53.562 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-12-16T20:44:53.562 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:53.562 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-12-16T20:44:53.562 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-12-16T20:44:53.562 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2023-12-16T20:44:53.562 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2023-12-16T20:44:53.562 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-12-16T20:44:53.562 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "smithi052" 2023-12-16T20:44:53.562 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2023-12-16T20:44:53.562 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-12-16T20:44:53.562 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2023-12-16T20:44:53.562 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:53.563 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-12-16T20:44:53.563 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-12-16T20:44:53.563 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:53.563 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-12-16T20:44:53.563 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:53.563 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-12-16T20:44:53.563 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-12-16T20:44:53.563 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-12-16T20:44:53.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-12-16T20:44:53.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-12-16T20:44:53.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-12-16T20:44:53.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:53.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-12-16T20:44:53.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-12-16T20:44:53.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-12-16T20:44:53.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-12-16T20:44:53.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-12-16T20:44:53.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-12-16T20:44:53.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-12-16T20:44:53.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-12-16T20:44:53.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-12-16T20:44:53.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:53.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-12-16T20:44:53.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:53.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-12-16T20:44:53.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-12-16T20:44:53.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:53.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-12-16T20:44:53.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2023-12-16T20:44:53.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-12-16T20:44:53.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2023-12-16T20:44:53.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2023-12-16T20:44:53.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2023-12-16T20:44:53.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2023-12-16T20:44:53.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2023-12-16T20:44:53.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2023-12-16T20:44:44.680846+0000", 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.237+0000 7f5addb7a700 1 Processor -- start 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.237+0000 7f5addb7a700 1 -- start start 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.237+0000 7f5addb7a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ad8076c00 0x7f5ad81072f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.237+0000 7f5addb7a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad8102ce0 con 0x7f5ad8076c00 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.237+0000 7f5ad77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ad8076c00 0x7f5ad81072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.237+0000 7f5ad77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ad8076c00 0x7f5ad81072f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49922/0 (socket says 172.21.15.52:49922) 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.237+0000 7f5ad77fe700 1 -- 172.21.15.52:0/147703825 learned_addr learned my addr 172.21.15.52:0/147703825 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:53.573 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.237+0000 7f5ad77fe700 1 -- 172.21.15.52:0/147703825 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad8107830 con 0x7f5ad8076c00 2023-12-16T20:44:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.237+0000 7f5ad77fe700 1 --2- 172.21.15.52:0/147703825 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ad8076c00 0x7f5ad81072f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5ac80098e0 tx=0x7f5ac8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d7952449a2c7d6d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.238+0000 7f5ad67fc700 1 -- 172.21.15.52:0/147703825 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ac800b860 con 0x7f5ad8076c00 2023-12-16T20:44:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.238+0000 7f5ad67fc700 1 -- 172.21.15.52:0/147703825 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5ac800be30 con 0x7f5ad8076c00 2023-12-16T20:44:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.238+0000 7f5ad67fc700 1 -- 172.21.15.52:0/147703825 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ac8010380 con 0x7f5ad8076c00 2023-12-16T20:44:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.238+0000 7f5addb7a700 1 -- 172.21.15.52:0/147703825 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ad8076c00 msgr2=0x7f5ad81072f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.239+0000 7f5addb7a700 1 --2- 172.21.15.52:0/147703825 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ad8076c00 0x7f5ad81072f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5ac80098e0 tx=0x7f5ac8009250 comp rx=0 tx=0).stop 2023-12-16T20:44:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.239+0000 7f5addb7a700 1 -- 172.21.15.52:0/147703825 shutdown_connections 2023-12-16T20:44:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.239+0000 7f5addb7a700 1 --2- 172.21.15.52:0/147703825 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ad8076c00 0x7f5ad81072f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.239+0000 7f5addb7a700 1 -- 172.21.15.52:0/147703825 >> 172.21.15.52:0/147703825 conn(0x7f5ad80fc980 msgr2=0x7f5ad80fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.239+0000 7f5addb7a700 1 -- 172.21.15.52:0/147703825 shutdown_connections 2023-12-16T20:44:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.239+0000 7f5addb7a700 1 -- 172.21.15.52:0/147703825 wait complete. 2023-12-16T20:44:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.239+0000 7f5addb7a700 1 Processor -- start 2023-12-16T20:44:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.239+0000 7f5addb7a700 1 -- start start 2023-12-16T20:44:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.240+0000 7f5addb7a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ad8076c00 0x7f5ad8113370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.240+0000 7f5addb7a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad81138b0 con 0x7f5ad8076c00 2023-12-16T20:44:53.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.240+0000 7f5ad77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ad8076c00 0x7f5ad8113370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:53.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.240+0000 7f5ad77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ad8076c00 0x7f5ad8113370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49936/0 (socket says 172.21.15.52:49936) 2023-12-16T20:44:53.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.240+0000 7f5ad77fe700 1 -- 172.21.15.52:0/3174229735 learned_addr learned my addr 172.21.15.52:0/3174229735 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:53.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.240+0000 7f5ad77fe700 1 -- 172.21.15.52:0/3174229735 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac80094f0 con 0x7f5ad8076c00 2023-12-16T20:44:53.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.240+0000 7f5ad77fe700 1 --2- 172.21.15.52:0/3174229735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ad8076c00 0x7f5ad8113370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8006b00 tx=0x7f5ac8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:53.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.240+0000 7f5abffff700 1 -- 172.21.15.52:0/3174229735 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ac8010a80 con 0x7f5ad8076c00 2023-12-16T20:44:53.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.240+0000 7f5abffff700 1 -- 172.21.15.52:0/3174229735 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5ac801a430 con 0x7f5ad8076c00 2023-12-16T20:44:53.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.240+0000 7f5abffff700 1 -- 172.21.15.52:0/3174229735 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ac80245b0 con 0x7f5ad8076c00 2023-12-16T20:44:53.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.240+0000 7f5addb7a700 1 -- 172.21.15.52:0/3174229735 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad8113ab0 con 0x7f5ad8076c00 2023-12-16T20:44:53.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.241+0000 7f5addb7a700 1 -- 172.21.15.52:0/3174229735 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad8113ec0 con 0x7f5ad8076c00 2023-12-16T20:44:53.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.241+0000 7f5abffff700 1 -- 172.21.15.52:0/3174229735 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f5ac8024a60 con 0x7f5ad8076c00 2023-12-16T20:44:53.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.241+0000 7f5abffff700 1 -- 172.21.15.52:0/3174229735 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5ac802eaa0 con 0x7f5ad8076c00 2023-12-16T20:44:53.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.242+0000 7f5addb7a700 1 -- 172.21.15.52:0/3174229735 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad804bb80 con 0x7f5ad8076c00 2023-12-16T20:44:53.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.244+0000 7f5abffff700 1 -- 172.21.15.52:0/3174229735 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f5ac802ed00 con 0x7f5ad8076c00 2023-12-16T20:44:53.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.332+0000 7f5addb7a700 1 -- 172.21.15.52:0/3174229735 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5ad810cd50 con 0x7f5ad8076c00 2023-12-16T20:44:53.576 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.333+0000 7f5abffff700 1 -- 172.21.15.52:0/3174229735 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f5ac8010be0 con 0x7f5ad8076c00 2023-12-16T20:44:53.576 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.334+0000 7f5addb7a700 1 -- 172.21.15.52:0/3174229735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ad8076c00 msgr2=0x7f5ad8113370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:53.576 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.334+0000 7f5addb7a700 1 --2- 172.21.15.52:0/3174229735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ad8076c00 0x7f5ad8113370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8006b00 tx=0x7f5ac8009250 comp rx=0 tx=0).stop 2023-12-16T20:44:53.576 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.334+0000 7f5addb7a700 1 -- 172.21.15.52:0/3174229735 shutdown_connections 2023-12-16T20:44:53.576 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.334+0000 7f5addb7a700 1 --2- 172.21.15.52:0/3174229735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ad8076c00 0x7f5ad8113370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:53.576 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.334+0000 7f5addb7a700 1 -- 172.21.15.52:0/3174229735 >> 172.21.15.52:0/3174229735 conn(0x7f5ad80fc980 msgr2=0x7f5ad80fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:53.576 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.334+0000 7f5addb7a700 1 -- 172.21.15.52:0/3174229735 shutdown_connections 2023-12-16T20:44:53.576 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:53.334+0000 7f5addb7a700 1 -- 172.21.15.52:0/3174229735 wait complete. 2023-12-16T20:44:53.576 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (2/15)... 2023-12-16T20:44:53.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:53 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3174229735' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-16T20:44:55.865 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:55 smithi052 ceph-mon[80761]: Activating manager daemon smithi052.nmkcbs 2023-12-16T20:44:55.866 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:55 smithi052 ceph-mon[80761]: mgrmap e2: smithi052.nmkcbs(active, starting, since 0.00294565s) 2023-12-16T20:44:55.866 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:55 smithi052 ceph-mon[80761]: from='mgr.14100 172.21.15.52:0/1247170213' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T20:44:55.866 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:55 smithi052 ceph-mon[80761]: from='mgr.14100 172.21.15.52:0/1247170213' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T20:44:55.866 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:55 smithi052 ceph-mon[80761]: from='mgr.14100 172.21.15.52:0/1247170213' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T20:44:55.866 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:55 smithi052 ceph-mon[80761]: from='mgr.14100 172.21.15.52:0/1247170213' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2023-12-16T20:44:55.866 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:55 smithi052 ceph-mon[80761]: from='mgr.14100 172.21.15.52:0/1247170213' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr metadata", "who": "smithi052.nmkcbs", "id": "smithi052.nmkcbs"}]: dispatch 2023-12-16T20:44:55.866 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:55 smithi052 ceph-mon[80761]: Manager daemon smithi052.nmkcbs is now available 2023-12-16T20:44:55.866 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:55 smithi052 ceph-mon[80761]: from='mgr.14100 172.21.15.52:0/1247170213' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.nmkcbs/mirror_snapshot_schedule"}]: dispatch 2023-12-16T20:44:55.866 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:55 smithi052 ceph-mon[80761]: from='mgr.14100 172.21.15.52:0/1247170213' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.nmkcbs/trash_purge_schedule"}]: dispatch 2023-12-16T20:44:55.866 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:55 smithi052 ceph-mon[80761]: from='mgr.14100 172.21.15.52:0/1247170213' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:44:55.866 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:55 smithi052 ceph-mon[80761]: from='mgr.14100 172.21.15.52:0/1247170213' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:44:55.866 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:55 smithi052 ceph-mon[80761]: from='mgr.14100 172.21.15.52:0/1247170213' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:44:56.201 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2023-12-16T20:44:56.201 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2023-12-16T20:44:56.201 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "cf043a8a-9c53-11ee-95a5-87774f69a715", 2023-12-16T20:44:56.201 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2023-12-16T20:44:56.201 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-12-16T20:44:56.201 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-12-16T20:44:56.201 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-12-16T20:44:56.201 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "smithi052" 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-12-16T20:44:56.203 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-12-16T20:44:56.203 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-12-16T20:44:56.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-12-16T20:44:56.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-12-16T20:44:56.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-12-16T20:44:56.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:56.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-12-16T20:44:56.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-12-16T20:44:56.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-12-16T20:44:56.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-12-16T20:44:56.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-12-16T20:44:56.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-12-16T20:44:56.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-12-16T20:44:56.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-12-16T20:44:56.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-12-16T20:44:56.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:56.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-12-16T20:44:56.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:56.213 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2023-12-16T20:44:44.680846+0000", 2023-12-16T20:44:56.213 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T20:44:56.213 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:56.213 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-12-16T20:44:56.213 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2023-12-16T20:44:56.213 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.784+0000 7fbe07fe3700 1 Processor -- start 2023-12-16T20:44:56.213 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.784+0000 7fbe07fe3700 1 -- start start 2023-12-16T20:44:56.213 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.784+0000 7fbe07fe3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe001039c0 0x7fbe00103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:56.213 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.784+0000 7fbe07fe3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe00104310 con 0x7fbe001039c0 2023-12-16T20:44:56.213 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.784+0000 7fbe05d7f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe001039c0 0x7fbe00103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:56.213 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.784+0000 7fbe05d7f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe001039c0 0x7fbe00103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49994/0 (socket says 172.21.15.52:49994) 2023-12-16T20:44:56.213 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.784+0000 7fbe05d7f700 1 -- 172.21.15.52:0/3884829879 learned_addr learned my addr 172.21.15.52:0/3884829879 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:56.213 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.785+0000 7fbe05d7f700 1 -- 172.21.15.52:0/3884829879 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe00101fb0 con 0x7fbe001039c0 2023-12-16T20:44:56.213 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.785+0000 7fbe05d7f700 1 --2- 172.21.15.52:0/3884829879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe001039c0 0x7fbe00103dd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc0098e0 tx=0x7fbdfc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf3b2d4c8f50ee5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:56.214 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.785+0000 7fbe04d7d700 1 -- 172.21.15.52:0/3884829879 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdfc00b860 con 0x7fbe001039c0 2023-12-16T20:44:56.214 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.785+0000 7fbe04d7d700 1 -- 172.21.15.52:0/3884829879 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fbdfc00be30 con 0x7fbe001039c0 2023-12-16T20:44:56.214 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.785+0000 7fbe04d7d700 1 -- 172.21.15.52:0/3884829879 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdfc010380 con 0x7fbe001039c0 2023-12-16T20:44:56.214 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.785+0000 7fbe07fe3700 1 -- 172.21.15.52:0/3884829879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe001039c0 msgr2=0x7fbe00103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:56.214 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.785+0000 7fbe07fe3700 1 --2- 172.21.15.52:0/3884829879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe001039c0 0x7fbe00103dd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc0098e0 tx=0x7fbdfc009250 comp rx=0 tx=0).stop 2023-12-16T20:44:56.214 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.786+0000 7fbe07fe3700 1 -- 172.21.15.52:0/3884829879 shutdown_connections 2023-12-16T20:44:56.214 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.786+0000 7fbe07fe3700 1 --2- 172.21.15.52:0/3884829879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe001039c0 0x7fbe00103dd0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:56.214 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.786+0000 7fbe07fe3700 1 -- 172.21.15.52:0/3884829879 >> 172.21.15.52:0/3884829879 conn(0x7fbe000fda40 msgr2=0x7fbe000ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:56.214 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.786+0000 7fbe07fe3700 1 -- 172.21.15.52:0/3884829879 shutdown_connections 2023-12-16T20:44:56.214 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.786+0000 7fbe07fe3700 1 -- 172.21.15.52:0/3884829879 wait complete. 2023-12-16T20:44:56.214 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.786+0000 7fbe07fe3700 1 Processor -- start 2023-12-16T20:44:56.215 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.786+0000 7fbe07fe3700 1 -- start start 2023-12-16T20:44:56.215 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.786+0000 7fbe07fe3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe001039c0 0x7fbe001110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:56.215 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.786+0000 7fbe07fe3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe00111630 con 0x7fbe001039c0 2023-12-16T20:44:56.215 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.787+0000 7fbe05d7f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe001039c0 0x7fbe001110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:56.215 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.787+0000 7fbe05d7f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe001039c0 0x7fbe001110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50004/0 (socket says 172.21.15.52:50004) 2023-12-16T20:44:56.215 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.787+0000 7fbe05d7f700 1 -- 172.21.15.52:0/2850619364 learned_addr learned my addr 172.21.15.52:0/2850619364 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:56.215 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.787+0000 7fbe05d7f700 1 -- 172.21.15.52:0/2850619364 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdfc0094f0 con 0x7fbe001039c0 2023-12-16T20:44:56.215 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.787+0000 7fbe05d7f700 1 --2- 172.21.15.52:0/2850619364 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe001039c0 0x7fbe001110f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc000c00 tx=0x7fbdfc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:56.215 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.787+0000 7fbdf67fc700 1 -- 172.21.15.52:0/2850619364 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdfc010a80 con 0x7fbe001039c0 2023-12-16T20:44:56.215 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.787+0000 7fbdf67fc700 1 -- 172.21.15.52:0/2850619364 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fbdfc01a430 con 0x7fbe001039c0 2023-12-16T20:44:56.215 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.787+0000 7fbe07fe3700 1 -- 172.21.15.52:0/2850619364 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe00111830 con 0x7fbe001039c0 2023-12-16T20:44:56.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.787+0000 7fbdf67fc700 1 -- 172.21.15.52:0/2850619364 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdfc0245b0 con 0x7fbe001039c0 2023-12-16T20:44:56.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.787+0000 7fbe07fe3700 1 -- 172.21.15.52:0/2850619364 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe00111c40 con 0x7fbe001039c0 2023-12-16T20:44:56.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.788+0000 7fbdf67fc700 1 -- 172.21.15.52:0/2850619364 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 2) v1 ==== 43037+0+0 (secure 0 0 0) 0x7fbdfc024870 con 0x7fbe001039c0 2023-12-16T20:44:56.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.788+0000 7fbe07fe3700 1 -- 172.21.15.52:0/2850619364 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe00102d60 con 0x7fbe001039c0 2023-12-16T20:44:56.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.788+0000 7fbdf67fc700 1 -- 172.21.15.52:0/2850619364 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fbdfc04a1b0 con 0x7fbe001039c0 2023-12-16T20:44:56.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.790+0000 7fbdf67fc700 1 -- 172.21.15.52:0/2850619364 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fbdfc02e440 con 0x7fbe001039c0 2023-12-16T20:44:56.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.864+0000 7fbe07fe3700 1 -- 172.21.15.52:0/2850619364 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbe0010ad50 con 0x7fbe001039c0 2023-12-16T20:44:56.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.864+0000 7fbdf67fc700 1 -- 172.21.15.52:0/2850619364 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fbdfc018350 con 0x7fbe001039c0 2023-12-16T20:44:56.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.865+0000 7fbe07fe3700 1 -- 172.21.15.52:0/2850619364 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe001039c0 msgr2=0x7fbe001110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:56.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.865+0000 7fbe07fe3700 1 --2- 172.21.15.52:0/2850619364 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe001039c0 0x7fbe001110f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc000c00 tx=0x7fbdfc009250 comp rx=0 tx=0).stop 2023-12-16T20:44:56.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.865+0000 7fbe07fe3700 1 -- 172.21.15.52:0/2850619364 shutdown_connections 2023-12-16T20:44:56.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.865+0000 7fbe07fe3700 1 --2- 172.21.15.52:0/2850619364 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe001039c0 0x7fbe001110f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:56.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.865+0000 7fbe07fe3700 1 -- 172.21.15.52:0/2850619364 >> 172.21.15.52:0/2850619364 conn(0x7fbe000fda40 msgr2=0x7fbe000ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:56.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.865+0000 7fbe07fe3700 1 -- 172.21.15.52:0/2850619364 shutdown_connections 2023-12-16T20:44:56.217 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:55.865+0000 7fbe07fe3700 1 -- 172.21.15.52:0/2850619364 wait complete. 2023-12-16T20:44:56.217 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (3/15)... 2023-12-16T20:44:57.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:56 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/2850619364' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-16T20:44:57.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:56 smithi052 ceph-mon[80761]: mgrmap e3: smithi052.nmkcbs(active, since 1.00673s) 2023-12-16T20:44:59.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2023-12-16T20:44:59.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2023-12-16T20:44:59.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "cf043a8a-9c53-11ee-95a5-87774f69a715", 2023-12-16T20:44:59.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2023-12-16T20:44:59.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-12-16T20:44:59.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-12-16T20:44:59.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-12-16T20:44:59.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:59.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-12-16T20:44:59.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-12-16T20:44:59.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2023-12-16T20:44:59.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2023-12-16T20:44:59.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-12-16T20:44:59.022 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "smithi052" 2023-12-16T20:44:59.022 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2023-12-16T20:44:59.022 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2023-12-16T20:44:59.022 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2023-12-16T20:44:59.022 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:59.022 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-12-16T20:44:59.022 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-12-16T20:44:59.022 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:59.022 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-12-16T20:44:59.022 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:59.022 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-12-16T20:44:59.022 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-12-16T20:44:59.022 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-12-16T20:44:59.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-12-16T20:44:59.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-12-16T20:44:59.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-12-16T20:44:59.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:59.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-12-16T20:44:59.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-12-16T20:44:59.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-12-16T20:44:59.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-12-16T20:44:59.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-12-16T20:44:59.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-12-16T20:44:59.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-12-16T20:44:59.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-12-16T20:44:59.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-12-16T20:44:59.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:59.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-12-16T20:44:59.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:59.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-12-16T20:44:59.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-12-16T20:44:59.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:59.033 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-12-16T20:44:59.033 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2023-12-16T20:44:59.033 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-12-16T20:44:59.033 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2023-12-16T20:44:59.033 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2023-12-16T20:44:59.033 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2023-12-16T20:44:59.033 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2023-12-16T20:44:59.033 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2023-12-16T20:44:59.033 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T20:44:59.033 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:59.033 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-12-16T20:44:59.033 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T20:44:59.033 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2023-12-16T20:44:44.680846+0000", 2023-12-16T20:44:59.033 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T20:44:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2023-12-16T20:44:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-12-16T20:44:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2023-12-16T20:44:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.444+0000 7f05310b9700 1 Processor -- start 2023-12-16T20:44:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.444+0000 7f05310b9700 1 -- start start 2023-12-16T20:44:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.444+0000 7f05310b9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f052c1027b0 0x7f052c102bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.444+0000 7f05310b9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c103100 con 0x7f052c1027b0 2023-12-16T20:44:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.444+0000 7f052ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f052c1027b0 0x7f052c102bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.444+0000 7f052ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f052c1027b0 0x7f052c102bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50012/0 (socket says 172.21.15.52:50012) 2023-12-16T20:44:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.444+0000 7f052ad9d700 1 -- 172.21.15.52:0/3066613778 learned_addr learned my addr 172.21.15.52:0/3066613778 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.445+0000 7f052ad9d700 1 -- 172.21.15.52:0/3066613778 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c103240 con 0x7f052c1027b0 2023-12-16T20:44:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.445+0000 7f052ad9d700 1 --2- 172.21.15.52:0/3066613778 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f052c1027b0 0x7f052c102bc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f051c0098e0 tx=0x7f051c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=204539bcc09012b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.445+0000 7f0529d9b700 1 -- 172.21.15.52:0/3066613778 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f051c00b860 con 0x7f052c1027b0 2023-12-16T20:44:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.445+0000 7f0529d9b700 1 -- 172.21.15.52:0/3066613778 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f051c00be30 con 0x7f052c1027b0 2023-12-16T20:44:59.035 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.445+0000 7f0529d9b700 1 -- 172.21.15.52:0/3066613778 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f051c010380 con 0x7f052c1027b0 2023-12-16T20:44:59.035 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.446+0000 7f05310b9700 1 -- 172.21.15.52:0/3066613778 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f052c1027b0 msgr2=0x7f052c102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:59.035 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.446+0000 7f05310b9700 1 --2- 172.21.15.52:0/3066613778 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f052c1027b0 0x7f052c102bc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f051c0098e0 tx=0x7f051c009250 comp rx=0 tx=0).stop 2023-12-16T20:44:59.035 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.446+0000 7f05310b9700 1 -- 172.21.15.52:0/3066613778 shutdown_connections 2023-12-16T20:44:59.035 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.446+0000 7f05310b9700 1 --2- 172.21.15.52:0/3066613778 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f052c1027b0 0x7f052c102bc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:59.035 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.446+0000 7f05310b9700 1 -- 172.21.15.52:0/3066613778 >> 172.21.15.52:0/3066613778 conn(0x7f052c0fc0b0 msgr2=0x7f052c0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:59.035 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.446+0000 7f05310b9700 1 -- 172.21.15.52:0/3066613778 shutdown_connections 2023-12-16T20:44:59.035 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.446+0000 7f05310b9700 1 -- 172.21.15.52:0/3066613778 wait complete. 2023-12-16T20:44:59.035 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.446+0000 7f05310b9700 1 Processor -- start 2023-12-16T20:44:59.035 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.446+0000 7f05310b9700 1 -- start start 2023-12-16T20:44:59.035 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.447+0000 7f05310b9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f052c1027b0 0x7f052c111160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:59.035 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.447+0000 7f05310b9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c1116a0 con 0x7f052c1027b0 2023-12-16T20:44:59.035 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.447+0000 7f052ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f052c1027b0 0x7f052c111160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:59.035 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.447+0000 7f052ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f052c1027b0 0x7f052c111160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50024/0 (socket says 172.21.15.52:50024) 2023-12-16T20:44:59.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.447+0000 7f052ad9d700 1 -- 172.21.15.52:0/1516454670 learned_addr learned my addr 172.21.15.52:0/1516454670 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:59.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.447+0000 7f052ad9d700 1 -- 172.21.15.52:0/1516454670 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c0094f0 con 0x7f052c1027b0 2023-12-16T20:44:59.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.447+0000 7f052ad9d700 1 --2- 172.21.15.52:0/1516454670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f052c1027b0 0x7f052c111160 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f051c000c00 tx=0x7f051c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:59.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.447+0000 7f05137fe700 1 -- 172.21.15.52:0/1516454670 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f051c010a80 con 0x7f052c1027b0 2023-12-16T20:44:59.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.447+0000 7f05310b9700 1 -- 172.21.15.52:0/1516454670 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052c1118a0 con 0x7f052c1027b0 2023-12-16T20:44:59.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.447+0000 7f05137fe700 1 -- 172.21.15.52:0/1516454670 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f051c01a430 con 0x7f052c1027b0 2023-12-16T20:44:59.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.447+0000 7f05137fe700 1 -- 172.21.15.52:0/1516454670 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f051c0245b0 con 0x7f052c1027b0 2023-12-16T20:44:59.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.447+0000 7f05310b9700 1 -- 172.21.15.52:0/1516454670 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f052c111cb0 con 0x7f052c1027b0 2023-12-16T20:44:59.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.448+0000 7f05137fe700 1 -- 172.21.15.52:0/1516454670 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7f051c010be0 con 0x7f052c1027b0 2023-12-16T20:44:59.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.448+0000 7f05137fe700 1 --2- 172.21.15.52:0/1516454670 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f05140365a0 0x7f0514038a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:59.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.448+0000 7f05310b9700 1 -- 172.21.15.52:0/1516454670 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052c04bb80 con 0x7f052c1027b0 2023-12-16T20:44:59.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.449+0000 7f05137fe700 1 -- 172.21.15.52:0/1516454670 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f051c04aaf0 con 0x7f052c1027b0 2023-12-16T20:44:59.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.449+0000 7f052a59c700 1 --2- 172.21.15.52:0/1516454670 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f05140365a0 0x7f0514038a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:59.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.449+0000 7f052a59c700 1 --2- 172.21.15.52:0/1516454670 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f05140365a0 0x7f0514038a50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0520009990 tx=0x7f0520006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:59.037 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.454+0000 7f05137fe700 1 -- 172.21.15.52:0/1516454670 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f051c023390 con 0x7f052c1027b0 2023-12-16T20:44:59.037 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.727+0000 7f05310b9700 1 -- 172.21.15.52:0/1516454670 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f052c112920 con 0x7f052c1027b0 2023-12-16T20:44:59.037 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.728+0000 7f05137fe700 1 -- 172.21.15.52:0/1516454670 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f051c00fd90 con 0x7f052c1027b0 2023-12-16T20:44:59.037 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.730+0000 7f05310b9700 1 -- 172.21.15.52:0/1516454670 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f05140365a0 msgr2=0x7f0514038a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:59.037 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.730+0000 7f05310b9700 1 --2- 172.21.15.52:0/1516454670 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f05140365a0 0x7f0514038a50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0520009990 tx=0x7f0520006e40 comp rx=0 tx=0).stop 2023-12-16T20:44:59.037 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.730+0000 7f05310b9700 1 -- 172.21.15.52:0/1516454670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f052c1027b0 msgr2=0x7f052c111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:59.037 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.730+0000 7f05310b9700 1 --2- 172.21.15.52:0/1516454670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f052c1027b0 0x7f052c111160 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f051c000c00 tx=0x7f051c009250 comp rx=0 tx=0).stop 2023-12-16T20:44:59.037 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.730+0000 7f05310b9700 1 -- 172.21.15.52:0/1516454670 shutdown_connections 2023-12-16T20:44:59.037 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.730+0000 7f05310b9700 1 --2- 172.21.15.52:0/1516454670 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f05140365a0 0x7f0514038a50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:59.037 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.730+0000 7f05310b9700 1 --2- 172.21.15.52:0/1516454670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f052c1027b0 0x7f052c111160 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:59.037 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.730+0000 7f05310b9700 1 -- 172.21.15.52:0/1516454670 >> 172.21.15.52:0/1516454670 conn(0x7f052c0fc0b0 msgr2=0x7f052c100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:59.037 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.730+0000 7f05310b9700 1 -- 172.21.15.52:0/1516454670 shutdown_connections 2023-12-16T20:44:59.037 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:58.730+0000 7f05310b9700 1 -- 172.21.15.52:0/1516454670 wait complete. 2023-12-16T20:44:59.038 INFO:teuthology.orchestra.run.smithi052.stdout:mgr is available 2023-12-16T20:44:59.572 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:59 smithi052 ceph-mon[80761]: mgrmap e4: smithi052.nmkcbs(active, since 2s) 2023-12-16T20:44:59.573 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:44:59 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/1516454670' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-16T20:44:59.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2023-12-16T20:44:59.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [global] 2023-12-16T20:44:59.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout fsid = cf043a8a-9c53-11ee-95a5-87774f69a715 2023-12-16T20:44:59.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [mgr] 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [osd] 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.244+0000 7fd0d148c700 1 Processor -- start 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.244+0000 7fd0d148c700 1 -- start start 2023-12-16T20:44:59.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.244+0000 7fd0d148c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd0cc1039c0 0x7fd0cc103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:59.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.244+0000 7fd0d148c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc104310 con 0x7fd0cc1039c0 2023-12-16T20:44:59.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.244+0000 7fd0caffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd0cc1039c0 0x7fd0cc103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:59.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.244+0000 7fd0caffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd0cc1039c0 0x7fd0cc103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50030/0 (socket says 172.21.15.52:50030) 2023-12-16T20:44:59.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.244+0000 7fd0caffd700 1 -- 172.21.15.52:0/404956298 learned_addr learned my addr 172.21.15.52:0/404956298 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:59.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.244+0000 7fd0caffd700 1 -- 172.21.15.52:0/404956298 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0cc101fb0 con 0x7fd0cc1039c0 2023-12-16T20:44:59.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.244+0000 7fd0caffd700 1 --2- 172.21.15.52:0/404956298 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd0cc1039c0 0x7fd0cc103dd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd0c40098e0 tx=0x7fd0c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eede29e0bbcb9d3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:59.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.245+0000 7fd0c9ffb700 1 -- 172.21.15.52:0/404956298 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0c400b860 con 0x7fd0cc1039c0 2023-12-16T20:44:59.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.245+0000 7fd0c9ffb700 1 -- 172.21.15.52:0/404956298 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd0c400be30 con 0x7fd0cc1039c0 2023-12-16T20:44:59.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.245+0000 7fd0c9ffb700 1 -- 172.21.15.52:0/404956298 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0c4010380 con 0x7fd0cc1039c0 2023-12-16T20:44:59.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.245+0000 7fd0d148c700 1 -- 172.21.15.52:0/404956298 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd0cc1039c0 msgr2=0x7fd0cc103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:59.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.245+0000 7fd0d148c700 1 --2- 172.21.15.52:0/404956298 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd0cc1039c0 0x7fd0cc103dd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd0c40098e0 tx=0x7fd0c4009250 comp rx=0 tx=0).stop 2023-12-16T20:44:59.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.245+0000 7fd0d148c700 1 -- 172.21.15.52:0/404956298 shutdown_connections 2023-12-16T20:44:59.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.245+0000 7fd0d148c700 1 --2- 172.21.15.52:0/404956298 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd0cc1039c0 0x7fd0cc103dd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:59.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.245+0000 7fd0d148c700 1 -- 172.21.15.52:0/404956298 >> 172.21.15.52:0/404956298 conn(0x7fd0cc0fda40 msgr2=0x7fd0cc0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:59.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.245+0000 7fd0d148c700 1 -- 172.21.15.52:0/404956298 shutdown_connections 2023-12-16T20:44:59.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.245+0000 7fd0d148c700 1 -- 172.21.15.52:0/404956298 wait complete. 2023-12-16T20:44:59.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.246+0000 7fd0d148c700 1 Processor -- start 2023-12-16T20:44:59.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.246+0000 7fd0d148c700 1 -- start start 2023-12-16T20:44:59.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.246+0000 7fd0d148c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd0cc1039c0 0x7fd0cc1110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:59.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.246+0000 7fd0d148c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc111630 con 0x7fd0cc1039c0 2023-12-16T20:44:59.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.246+0000 7fd0caffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd0cc1039c0 0x7fd0cc1110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:59.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.246+0000 7fd0caffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd0cc1039c0 0x7fd0cc1110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50040/0 (socket says 172.21.15.52:50040) 2023-12-16T20:44:59.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.246+0000 7fd0caffd700 1 -- 172.21.15.52:0/3203113507 learned_addr learned my addr 172.21.15.52:0/3203113507 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:44:59.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.247+0000 7fd0caffd700 1 -- 172.21.15.52:0/3203113507 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c40094f0 con 0x7fd0cc1039c0 2023-12-16T20:44:59.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.247+0000 7fd0caffd700 1 --2- 172.21.15.52:0/3203113507 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd0cc1039c0 0x7fd0cc1110f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4006b00 tx=0x7fd0c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:59.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.247+0000 7fd0c37fe700 1 -- 172.21.15.52:0/3203113507 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0c40109b0 con 0x7fd0cc1039c0 2023-12-16T20:44:59.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.247+0000 7fd0c37fe700 1 -- 172.21.15.52:0/3203113507 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd0c401a430 con 0x7fd0cc1039c0 2023-12-16T20:44:59.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.247+0000 7fd0d148c700 1 -- 172.21.15.52:0/3203113507 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0cc111830 con 0x7fd0cc1039c0 2023-12-16T20:44:59.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.247+0000 7fd0c37fe700 1 -- 172.21.15.52:0/3203113507 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0c40245b0 con 0x7fd0cc1039c0 2023-12-16T20:44:59.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.247+0000 7fd0d148c700 1 -- 172.21.15.52:0/3203113507 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0cc111c40 con 0x7fd0cc1039c0 2023-12-16T20:44:59.693 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.248+0000 7fd0c37fe700 1 -- 172.21.15.52:0/3203113507 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7fd0c4024830 con 0x7fd0cc1039c0 2023-12-16T20:44:59.693 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.248+0000 7fd0c37fe700 1 --2- 172.21.15.52:0/3203113507 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fd0b80365a0 0x7fd0b8038a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:44:59.693 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.248+0000 7fd0c37fe700 1 -- 172.21.15.52:0/3203113507 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd0c404abc0 con 0x7fd0cc1039c0 2023-12-16T20:44:59.693 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.248+0000 7fd0d148c700 1 -- 172.21.15.52:0/3203113507 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0cc102d60 con 0x7fd0cc1039c0 2023-12-16T20:44:59.693 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.248+0000 7fd0ca7fc700 1 --2- 172.21.15.52:0/3203113507 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fd0b80365a0 0x7fd0b8038a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:44:59.693 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.249+0000 7fd0ca7fc700 1 --2- 172.21.15.52:0/3203113507 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fd0b80365a0 0x7fd0b8038a50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4009990 tx=0x7fd0b4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:44:59.693 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.253+0000 7fd0c37fe700 1 -- 172.21.15.52:0/3203113507 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd0c4024b30 con 0x7fd0cc1039c0 2023-12-16T20:44:59.693 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.447+0000 7fd0d148c700 1 -- 172.21.15.52:0/3203113507 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fd0cc04bb80 con 0x7fd0cc1039c0 2023-12-16T20:44:59.693 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.448+0000 7fd0c37fe700 1 -- 172.21.15.52:0/3203113507 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fd0c402cb30 con 0x7fd0cc1039c0 2023-12-16T20:44:59.693 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.450+0000 7fd0d148c700 1 -- 172.21.15.52:0/3203113507 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fd0b80365a0 msgr2=0x7fd0b8038a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:59.693 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.450+0000 7fd0d148c700 1 --2- 172.21.15.52:0/3203113507 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fd0b80365a0 0x7fd0b8038a50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4009990 tx=0x7fd0b4006e40 comp rx=0 tx=0).stop 2023-12-16T20:44:59.693 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.450+0000 7fd0d148c700 1 -- 172.21.15.52:0/3203113507 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd0cc1039c0 msgr2=0x7fd0cc1110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:44:59.693 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.450+0000 7fd0d148c700 1 --2- 172.21.15.52:0/3203113507 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd0cc1039c0 0x7fd0cc1110f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4006b00 tx=0x7fd0c4009250 comp rx=0 tx=0).stop 2023-12-16T20:44:59.694 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.450+0000 7fd0d148c700 1 -- 172.21.15.52:0/3203113507 shutdown_connections 2023-12-16T20:44:59.694 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.450+0000 7fd0d148c700 1 --2- 172.21.15.52:0/3203113507 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fd0b80365a0 0x7fd0b8038a50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:59.694 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.450+0000 7fd0d148c700 1 --2- 172.21.15.52:0/3203113507 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd0cc1039c0 0x7fd0cc1110f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:44:59.694 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.450+0000 7fd0d148c700 1 -- 172.21.15.52:0/3203113507 >> 172.21.15.52:0/3203113507 conn(0x7fd0cc0fda40 msgr2=0x7fd0cc0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:44:59.694 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.450+0000 7fd0d148c700 1 -- 172.21.15.52:0/3203113507 shutdown_connections 2023-12-16T20:44:59.694 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.450+0000 7fd0d148c700 1 -- 172.21.15.52:0/3203113507 wait complete. 2023-12-16T20:44:59.694 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling cephadm module... 2023-12-16T20:45:00.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:00 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3203113507' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-16T20:45:00.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:00 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/2127651366' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-12-16T20:45:00.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.935+0000 7f9e8094a700 1 Processor -- start 2023-12-16T20:45:00.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.936+0000 7f9e8094a700 1 -- start start 2023-12-16T20:45:00.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.936+0000 7f9e8094a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e7c1039c0 0x7f9e7c103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:00.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.936+0000 7f9e8094a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c104310 con 0x7f9e7c1039c0 2023-12-16T20:45:00.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.936+0000 7f9e7a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e7c1039c0 0x7f9e7c103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:00.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.936+0000 7f9e7a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e7c1039c0 0x7f9e7c103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50048/0 (socket says 172.21.15.52:50048) 2023-12-16T20:45:00.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.936+0000 7f9e7a59c700 1 -- 172.21.15.52:0/2704384469 learned_addr learned my addr 172.21.15.52:0/2704384469 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:00.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.936+0000 7f9e7a59c700 1 -- 172.21.15.52:0/2704384469 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c101fb0 con 0x7f9e7c1039c0 2023-12-16T20:45:00.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.936+0000 7f9e7a59c700 1 --2- 172.21.15.52:0/2704384469 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e7c1039c0 0x7f9e7c103dd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9e740098e0 tx=0x7f9e74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ce1ba863be02f6 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:00.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.937+0000 7f9e7959a700 1 -- 172.21.15.52:0/2704384469 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e7400b860 con 0x7f9e7c1039c0 2023-12-16T20:45:00.631 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.937+0000 7f9e7959a700 1 -- 172.21.15.52:0/2704384469 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f9e7400be30 con 0x7f9e7c1039c0 2023-12-16T20:45:00.631 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.937+0000 7f9e7959a700 1 -- 172.21.15.52:0/2704384469 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e74010380 con 0x7f9e7c1039c0 2023-12-16T20:45:00.631 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.937+0000 7f9e8094a700 1 -- 172.21.15.52:0/2704384469 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e7c1039c0 msgr2=0x7f9e7c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:00.631 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.937+0000 7f9e8094a700 1 --2- 172.21.15.52:0/2704384469 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e7c1039c0 0x7f9e7c103dd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9e740098e0 tx=0x7f9e74009250 comp rx=0 tx=0).stop 2023-12-16T20:45:00.631 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.937+0000 7f9e8094a700 1 -- 172.21.15.52:0/2704384469 shutdown_connections 2023-12-16T20:45:00.631 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.937+0000 7f9e8094a700 1 --2- 172.21.15.52:0/2704384469 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e7c1039c0 0x7f9e7c103dd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:00.631 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.937+0000 7f9e8094a700 1 -- 172.21.15.52:0/2704384469 >> 172.21.15.52:0/2704384469 conn(0x7f9e7c0fda40 msgr2=0x7f9e7c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:00.631 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.937+0000 7f9e8094a700 1 -- 172.21.15.52:0/2704384469 shutdown_connections 2023-12-16T20:45:00.631 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.937+0000 7f9e8094a700 1 -- 172.21.15.52:0/2704384469 wait complete. 2023-12-16T20:45:00.631 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.938+0000 7f9e8094a700 1 Processor -- start 2023-12-16T20:45:00.631 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.938+0000 7f9e8094a700 1 -- start start 2023-12-16T20:45:00.631 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.938+0000 7f9e8094a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e7c1039c0 0x7f9e7c1110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:00.631 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.938+0000 7f9e8094a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c111630 con 0x7f9e7c1039c0 2023-12-16T20:45:00.631 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.938+0000 7f9e7a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e7c1039c0 0x7f9e7c1110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:00.632 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.938+0000 7f9e7a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e7c1039c0 0x7f9e7c1110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50054/0 (socket says 172.21.15.52:50054) 2023-12-16T20:45:00.632 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.938+0000 7f9e7a59c700 1 -- 172.21.15.52:0/2127651366 learned_addr learned my addr 172.21.15.52:0/2127651366 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:00.632 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.938+0000 7f9e7a59c700 1 -- 172.21.15.52:0/2127651366 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e740094f0 con 0x7f9e7c1039c0 2023-12-16T20:45:00.632 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.939+0000 7f9e7a59c700 1 --2- 172.21.15.52:0/2127651366 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e7c1039c0 0x7f9e7c1110f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9e74000c00 tx=0x7f9e74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:00.632 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.939+0000 7f9e72ffd700 1 -- 172.21.15.52:0/2127651366 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e74010a80 con 0x7f9e7c1039c0 2023-12-16T20:45:00.632 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.939+0000 7f9e72ffd700 1 -- 172.21.15.52:0/2127651366 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f9e7401a430 con 0x7f9e7c1039c0 2023-12-16T20:45:00.632 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.939+0000 7f9e8094a700 1 -- 172.21.15.52:0/2127651366 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7c111830 con 0x7f9e7c1039c0 2023-12-16T20:45:00.632 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.939+0000 7f9e72ffd700 1 -- 172.21.15.52:0/2127651366 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e740245b0 con 0x7f9e7c1039c0 2023-12-16T20:45:00.632 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.939+0000 7f9e8094a700 1 -- 172.21.15.52:0/2127651366 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7c111c40 con 0x7f9e7c1039c0 2023-12-16T20:45:00.632 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.940+0000 7f9e72ffd700 1 -- 172.21.15.52:0/2127651366 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7f9e74024870 con 0x7f9e7c1039c0 2023-12-16T20:45:00.632 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.940+0000 7f9e72ffd700 1 --2- 172.21.15.52:0/2127651366 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f9e680365a0 0x7f9e68038a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:00.632 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.940+0000 7f9e8094a700 1 -- 172.21.15.52:0/2127651366 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7c102d60 con 0x7f9e7c1039c0 2023-12-16T20:45:00.632 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.940+0000 7f9e72ffd700 1 -- 172.21.15.52:0/2127651366 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9e7404a820 con 0x7f9e7c1039c0 2023-12-16T20:45:00.632 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.940+0000 7f9e79d9b700 1 --2- 172.21.15.52:0/2127651366 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f9e680365a0 0x7f9e68038a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:00.632 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.940+0000 7f9e79d9b700 1 --2- 172.21.15.52:0/2127651366 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f9e680365a0 0x7f9e68038a50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009990 tx=0x7f9e64006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:00.633 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:44:59.945+0000 7f9e72ffd700 1 -- 172.21.15.52:0/2127651366 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9e7402c340 con 0x7f9e7c1039c0 2023-12-16T20:45:00.633 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.198+0000 7f9e8094a700 1 -- 172.21.15.52:0/2127651366 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f9e7c04bb80 con 0x7f9e7c1039c0 2023-12-16T20:45:00.633 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.390+0000 7f9e72ffd700 1 -- 172.21.15.52:0/2127651366 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f9e7402e960 con 0x7f9e7c1039c0 2023-12-16T20:45:00.633 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.391+0000 7f9e72ffd700 1 -- 172.21.15.52:0/2127651366 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f9e7400f540 con 0x7f9e7c1039c0 2023-12-16T20:45:00.633 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.394+0000 7f9e8094a700 1 -- 172.21.15.52:0/2127651366 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f9e680365a0 msgr2=0x7f9e68038a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:00.633 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.394+0000 7f9e8094a700 1 --2- 172.21.15.52:0/2127651366 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f9e680365a0 0x7f9e68038a50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009990 tx=0x7f9e64006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:00.633 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.394+0000 7f9e8094a700 1 -- 172.21.15.52:0/2127651366 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e7c1039c0 msgr2=0x7f9e7c1110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:00.633 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.394+0000 7f9e8094a700 1 --2- 172.21.15.52:0/2127651366 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e7c1039c0 0x7f9e7c1110f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9e74000c00 tx=0x7f9e74009250 comp rx=0 tx=0).stop 2023-12-16T20:45:00.633 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.394+0000 7f9e8094a700 1 -- 172.21.15.52:0/2127651366 shutdown_connections 2023-12-16T20:45:00.633 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.394+0000 7f9e8094a700 1 --2- 172.21.15.52:0/2127651366 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f9e680365a0 0x7f9e68038a50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:00.633 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.394+0000 7f9e8094a700 1 --2- 172.21.15.52:0/2127651366 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e7c1039c0 0x7f9e7c1110f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:00.633 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.394+0000 7f9e8094a700 1 -- 172.21.15.52:0/2127651366 >> 172.21.15.52:0/2127651366 conn(0x7f9e7c0fda40 msgr2=0x7f9e7c0ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:00.633 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.394+0000 7f9e8094a700 1 -- 172.21.15.52:0/2127651366 shutdown_connections 2023-12-16T20:45:00.633 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.394+0000 7f9e8094a700 1 -- 172.21.15.52:0/2127651366 wait complete. 2023-12-16T20:45:01.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2023-12-16T20:45:01.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-12-16T20:45:01.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2023-12-16T20:45:01.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "active_name": "smithi052.nmkcbs", 2023-12-16T20:45:01.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-12-16T20:45:01.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2023-12-16T20:45:01.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.852+0000 7f308f5d6700 1 Processor -- start 2023-12-16T20:45:01.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.852+0000 7f308f5d6700 1 -- start start 2023-12-16T20:45:01.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.852+0000 7f308f5d6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30881011d0 0x7f30881015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:01.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.852+0000 7f308f5d6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3088101b20 con 0x7f30881011d0 2023-12-16T20:45:01.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.852+0000 7f308d372700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30881011d0 0x7f30881015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:01.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.853+0000 7f308d372700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30881011d0 0x7f30881015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50074/0 (socket says 172.21.15.52:50074) 2023-12-16T20:45:01.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.853+0000 7f308d372700 1 -- 172.21.15.52:0/263033547 learned_addr learned my addr 172.21.15.52:0/263033547 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:01.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.853+0000 7f308d372700 1 -- 172.21.15.52:0/263033547 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3088101c60 con 0x7f30881011d0 2023-12-16T20:45:01.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.853+0000 7f308d372700 1 --2- 172.21.15.52:0/263033547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30881011d0 0x7f30881015e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f307800bb80 tx=0x7f307800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13812d94bdd3049 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:01.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.853+0000 7f307ffff700 1 -- 172.21.15.52:0/263033547 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f307800dd00 con 0x7f30881011d0 2023-12-16T20:45:01.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.853+0000 7f307ffff700 1 -- 172.21.15.52:0/263033547 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f307800de60 con 0x7f30881011d0 2023-12-16T20:45:01.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.853+0000 7f307ffff700 1 -- 172.21.15.52:0/263033547 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3078018840 con 0x7f30881011d0 2023-12-16T20:45:01.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.854+0000 7f308f5d6700 1 -- 172.21.15.52:0/263033547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30881011d0 msgr2=0x7f30881015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:01.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.854+0000 7f308f5d6700 1 --2- 172.21.15.52:0/263033547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30881011d0 0x7f30881015e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f307800bb80 tx=0x7f307800b4f0 comp rx=0 tx=0).stop 2023-12-16T20:45:01.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.854+0000 7f308f5d6700 1 -- 172.21.15.52:0/263033547 shutdown_connections 2023-12-16T20:45:01.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.854+0000 7f308f5d6700 1 --2- 172.21.15.52:0/263033547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30881011d0 0x7f30881015e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:01.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.854+0000 7f308f5d6700 1 -- 172.21.15.52:0/263033547 >> 172.21.15.52:0/263033547 conn(0x7f30880fc970 msgr2=0x7f30880fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:01.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.854+0000 7f308f5d6700 1 -- 172.21.15.52:0/263033547 shutdown_connections 2023-12-16T20:45:01.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.854+0000 7f308f5d6700 1 -- 172.21.15.52:0/263033547 wait complete. 2023-12-16T20:45:01.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.854+0000 7f308f5d6700 1 Processor -- start 2023-12-16T20:45:01.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.854+0000 7f308f5d6700 1 -- start start 2023-12-16T20:45:01.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.855+0000 7f308f5d6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30881011d0 0x7f308810ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:01.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.855+0000 7f308f5d6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308810f450 con 0x7f30881011d0 2023-12-16T20:45:01.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.855+0000 7f308d372700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30881011d0 0x7f308810ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:01.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.855+0000 7f308d372700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30881011d0 0x7f308810ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50078/0 (socket says 172.21.15.52:50078) 2023-12-16T20:45:01.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.855+0000 7f308d372700 1 -- 172.21.15.52:0/2638375897 learned_addr learned my addr 172.21.15.52:0/2638375897 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:01.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.855+0000 7f308d372700 1 -- 172.21.15.52:0/2638375897 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307800b790 con 0x7f30881011d0 2023-12-16T20:45:01.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.855+0000 7f308d372700 1 --2- 172.21.15.52:0/2638375897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30881011d0 0x7f308810ef10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f307800d650 tx=0x7f307800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:01.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.855+0000 7f307dffb700 1 -- 172.21.15.52:0/2638375897 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30780108e0 con 0x7f30881011d0 2023-12-16T20:45:01.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.855+0000 7f307dffb700 1 -- 172.21.15.52:0/2638375897 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f307801d050 con 0x7f30881011d0 2023-12-16T20:45:01.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.856+0000 7f308f5d6700 1 -- 172.21.15.52:0/2638375897 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308810f650 con 0x7f30881011d0 2023-12-16T20:45:01.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.856+0000 7f307dffb700 1 -- 172.21.15.52:0/2638375897 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3078022590 con 0x7f30881011d0 2023-12-16T20:45:01.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.856+0000 7f308f5d6700 1 -- 172.21.15.52:0/2638375897 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f308810fa60 con 0x7f30881011d0 2023-12-16T20:45:01.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.856+0000 7f307dffb700 1 -- 172.21.15.52:0/2638375897 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f3078022780 con 0x7f30881011d0 2023-12-16T20:45:01.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.857+0000 7f307dffb700 1 --2- 172.21.15.52:0/2638375897 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f3074036640 0x7f3074038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:01.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.857+0000 7f307dffb700 1 -- 172.21.15.52:0/2638375897 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3078014070 con 0x7f30881011d0 2023-12-16T20:45:01.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.857+0000 7f308cb71700 1 -- 172.21.15.52:0/2638375897 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f3074036640 msgr2=0x7f3074038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2740978054 2023-12-16T20:45:01.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.857+0000 7f308cb71700 1 --2- 172.21.15.52:0/2638375897 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f3074036640 0x7f3074038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T20:45:01.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.857+0000 7f308f5d6700 1 -- 172.21.15.52:0/2638375897 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308804bb80 con 0x7f30881011d0 2023-12-16T20:45:01.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:00.866+0000 7f307dffb700 1 -- 172.21.15.52:0/2638375897 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f307800fd60 con 0x7f30881011d0 2023-12-16T20:45:01.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.058+0000 7f308cb71700 1 -- 172.21.15.52:0/2638375897 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f3074036640 msgr2=0x7f3074038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2740978054 2023-12-16T20:45:01.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.058+0000 7f308cb71700 1 --2- 172.21.15.52:0/2638375897 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f3074036640 0x7f3074038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T20:45:01.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.133+0000 7f308f5d6700 1 -- 172.21.15.52:0/2638375897 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f30881088e0 con 0x7f30881011d0 2023-12-16T20:45:01.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.133+0000 7f307dffb700 1 -- 172.21.15.52:0/2638375897 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f307800fb80 con 0x7f30881011d0 2023-12-16T20:45:01.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.135+0000 7f308f5d6700 1 -- 172.21.15.52:0/2638375897 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f3074036640 msgr2=0x7f3074038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T20:45:01.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.135+0000 7f308f5d6700 1 --2- 172.21.15.52:0/2638375897 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f3074036640 0x7f3074038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:01.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.135+0000 7f308f5d6700 1 -- 172.21.15.52:0/2638375897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30881011d0 msgr2=0x7f308810ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:01.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.135+0000 7f308f5d6700 1 --2- 172.21.15.52:0/2638375897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30881011d0 0x7f308810ef10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f307800d650 tx=0x7f307800b4f0 comp rx=0 tx=0).stop 2023-12-16T20:45:01.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.135+0000 7f308f5d6700 1 -- 172.21.15.52:0/2638375897 shutdown_connections 2023-12-16T20:45:01.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.135+0000 7f308f5d6700 1 --2- 172.21.15.52:0/2638375897 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7f3074036640 0x7f3074038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:01.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.135+0000 7f308f5d6700 1 --2- 172.21.15.52:0/2638375897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30881011d0 0x7f308810ef10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:01.376 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.135+0000 7f308f5d6700 1 -- 172.21.15.52:0/2638375897 >> 172.21.15.52:0/2638375897 conn(0x7f30880fc970 msgr2=0x7f3088105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:01.376 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.136+0000 7f308f5d6700 1 -- 172.21.15.52:0/2638375897 shutdown_connections 2023-12-16T20:45:01.376 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.136+0000 7f308f5d6700 1 -- 172.21.15.52:0/2638375897 wait complete. 2023-12-16T20:45:01.376 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for the mgr to restart... 2023-12-16T20:45:01.376 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr epoch 5... 2023-12-16T20:45:01.732 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:01 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/2127651366' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-12-16T20:45:01.732 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:01 smithi052 ceph-mon[80761]: mgrmap e5: smithi052.nmkcbs(active, since 4s) 2023-12-16T20:45:01.733 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:01 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/2638375897' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-16T20:45:07.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:07 smithi052 ceph-mon[80761]: Active manager daemon smithi052.nmkcbs restarted 2023-12-16T20:45:07.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:07 smithi052 ceph-mon[80761]: Activating manager daemon smithi052.nmkcbs 2023-12-16T20:45:07.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:07 smithi052 ceph-mon[80761]: osdmap e2: 0 total, 0 up, 0 in 2023-12-16T20:45:07.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:07 smithi052 ceph-mon[80761]: mgrmap e6: smithi052.nmkcbs(active, starting, since 0.00312763s) 2023-12-16T20:45:07.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:07 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2023-12-16T20:45:07.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:07 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr metadata", "who": "smithi052.nmkcbs", "id": "smithi052.nmkcbs"}]: dispatch 2023-12-16T20:45:07.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:07 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T20:45:07.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:07 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T20:45:07.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:07 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T20:45:07.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:07 smithi052 ceph-mon[80761]: Manager daemon smithi052.nmkcbs is now available 2023-12-16T20:45:07.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:07 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:07.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:07 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:07.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:07 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:45:07.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:07 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:45:07.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:07 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:45:08.442 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2023-12-16T20:45:08.442 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-12-16T20:45:08.442 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "initialized": true 2023-12-16T20:45:08.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2023-12-16T20:45:08.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.596+0000 7fac2b9c7700 1 Processor -- start 2023-12-16T20:45:08.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.596+0000 7fac2b9c7700 1 -- start start 2023-12-16T20:45:08.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.596+0000 7fac2b9c7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac24070f80 0x7fac24104f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:08.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.596+0000 7fac2b9c7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac24105460 con 0x7fac24070f80 2023-12-16T20:45:08.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.596+0000 7fac29763700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac24070f80 0x7fac24104f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:08.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.596+0000 7fac29763700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac24070f80 0x7fac24104f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50088/0 (socket says 172.21.15.52:50088) 2023-12-16T20:45:08.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.596+0000 7fac29763700 1 -- 172.21.15.52:0/3899605411 learned_addr learned my addr 172.21.15.52:0/3899605411 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:08.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.597+0000 7fac29763700 1 -- 172.21.15.52:0/3899605411 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac241055a0 con 0x7fac24070f80 2023-12-16T20:45:08.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.597+0000 7fac29763700 1 --2- 172.21.15.52:0/3899605411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac24070f80 0x7fac24104f20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fac200098e0 tx=0x7fac20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b2fbf3aaac2abb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:08.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.597+0000 7fac1bfff700 1 -- 172.21.15.52:0/3899605411 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac2000b860 con 0x7fac24070f80 2023-12-16T20:45:08.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.597+0000 7fac1bfff700 1 -- 172.21.15.52:0/3899605411 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fac2000be30 con 0x7fac24070f80 2023-12-16T20:45:08.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.597+0000 7fac1bfff700 1 -- 172.21.15.52:0/3899605411 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac20010380 con 0x7fac24070f80 2023-12-16T20:45:08.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.598+0000 7fac2b9c7700 1 -- 172.21.15.52:0/3899605411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac24070f80 msgr2=0x7fac24104f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:08.444 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.598+0000 7fac2b9c7700 1 --2- 172.21.15.52:0/3899605411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac24070f80 0x7fac24104f20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fac200098e0 tx=0x7fac20009250 comp rx=0 tx=0).stop 2023-12-16T20:45:08.444 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.598+0000 7fac2b9c7700 1 -- 172.21.15.52:0/3899605411 shutdown_connections 2023-12-16T20:45:08.444 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.598+0000 7fac2b9c7700 1 --2- 172.21.15.52:0/3899605411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac24070f80 0x7fac24104f20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:08.444 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.598+0000 7fac2b9c7700 1 -- 172.21.15.52:0/3899605411 >> 172.21.15.52:0/3899605411 conn(0x7fac240fc970 msgr2=0x7fac240fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:08.444 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.598+0000 7fac2b9c7700 1 -- 172.21.15.52:0/3899605411 shutdown_connections 2023-12-16T20:45:08.444 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.598+0000 7fac2b9c7700 1 -- 172.21.15.52:0/3899605411 wait complete. 2023-12-16T20:45:08.444 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.598+0000 7fac2b9c7700 1 Processor -- start 2023-12-16T20:45:08.444 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.598+0000 7fac2b9c7700 1 -- start start 2023-12-16T20:45:08.444 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.599+0000 7fac2b9c7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac24070f80 0x7fac241029a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:08.444 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.599+0000 7fac2b9c7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac24100e20 con 0x7fac24070f80 2023-12-16T20:45:08.444 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.599+0000 7fac29763700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac24070f80 0x7fac241029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:08.444 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.599+0000 7fac29763700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac24070f80 0x7fac241029a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50092/0 (socket says 172.21.15.52:50092) 2023-12-16T20:45:08.444 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.599+0000 7fac29763700 1 -- 172.21.15.52:0/3336929389 learned_addr learned my addr 172.21.15.52:0/3336929389 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:08.444 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.599+0000 7fac29763700 1 -- 172.21.15.52:0/3336929389 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac200094f0 con 0x7fac24070f80 2023-12-16T20:45:08.444 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.599+0000 7fac29763700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac24070f80 0x7fac241029a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fac20006b00 tx=0x7fac20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:08.445 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.599+0000 7fac19ffb700 1 -- 172.21.15.52:0/3336929389 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac20010a80 con 0x7fac24070f80 2023-12-16T20:45:08.445 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.599+0000 7fac2b9c7700 1 -- 172.21.15.52:0/3336929389 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac24101020 con 0x7fac24070f80 2023-12-16T20:45:08.445 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.599+0000 7fac19ffb700 1 -- 172.21.15.52:0/3336929389 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fac2001a430 con 0x7fac24070f80 2023-12-16T20:45:08.445 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.599+0000 7fac19ffb700 1 -- 172.21.15.52:0/3336929389 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac200245b0 con 0x7fac24070f80 2023-12-16T20:45:08.445 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.600+0000 7fac2b9c7700 1 -- 172.21.15.52:0/3336929389 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac24101430 con 0x7fac24070f80 2023-12-16T20:45:08.445 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.600+0000 7fac19ffb700 1 -- 172.21.15.52:0/3336929389 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fac20024830 con 0x7fac24070f80 2023-12-16T20:45:08.445 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.601+0000 7fac19ffb700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fac100365f0 0x7fac10038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:08.445 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.601+0000 7fac19ffb700 1 -- 172.21.15.52:0/3336929389 --> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fac100391b0 con 0x7fac100365f0 2023-12-16T20:45:08.445 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.601+0000 7fac19ffb700 1 -- 172.21.15.52:0/3336929389 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fac2004aaf0 con 0x7fac24070f80 2023-12-16T20:45:08.445 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.601+0000 7fac28f62700 1 -- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fac100365f0 msgr2=0x7fac10038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2740978054 2023-12-16T20:45:08.445 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.601+0000 7fac28f62700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fac100365f0 0x7fac10038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T20:45:08.445 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.802+0000 7fac28f62700 1 -- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fac100365f0 msgr2=0x7fac10038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2740978054 2023-12-16T20:45:08.445 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:01.802+0000 7fac28f62700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fac100365f0 0x7fac10038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T20:45:08.446 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:02.203+0000 7fac28f62700 1 -- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fac100365f0 msgr2=0x7fac10038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2740978054 2023-12-16T20:45:08.446 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:02.203+0000 7fac28f62700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fac100365f0 0x7fac10038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-16T20:45:08.446 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:03.004+0000 7fac28f62700 1 -- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fac100365f0 msgr2=0x7fac10038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2740978054 2023-12-16T20:45:08.446 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:03.004+0000 7fac28f62700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fac100365f0 0x7fac10038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-16T20:45:08.446 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:04.605+0000 7fac28f62700 1 -- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fac100365f0 msgr2=0x7fac10038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2740978054 2023-12-16T20:45:08.446 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:04.605+0000 7fac28f62700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fac100365f0 0x7fac10038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-16T20:45:08.446 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:07.199+0000 7fac19ffb700 1 -- 172.21.15.52:0/3336929389 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mgrmap(e 6) v1 ==== 43048+0+0 (secure 0 0 0) 0x7fac2002e960 con 0x7fac24070f80 2023-12-16T20:45:08.446 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:07.199+0000 7fac19ffb700 1 -- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fac100365f0 msgr2=0x7fac10038aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T20:45:08.446 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:07.200+0000 7fac19ffb700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fac100365f0 0x7fac10038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:08.446 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.201+0000 7fac19ffb700 1 -- 172.21.15.52:0/3336929389 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7fac20016d40 con 0x7fac24070f80 2023-12-16T20:45:08.446 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.201+0000 7fac19ffb700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fac10039f60 0x7fac1003c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:08.446 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.201+0000 7fac19ffb700 1 -- 172.21.15.52:0/3336929389 --> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fac100391b0 con 0x7fac10039f60 2023-12-16T20:45:08.446 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.201+0000 7fac28f62700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fac10039f60 0x7fac1003c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:08.446 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.202+0000 7fac28f62700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fac10039f60 0x7fac1003c340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fac140039d0 tx=0x7fac14009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.203+0000 7fac19ffb700 1 -- 172.21.15.52:0/3336929389 <== mgr.14120 v2:172.21.15.52:6800/221719009 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fac100391b0 con 0x7fac10039f60 2023-12-16T20:45:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.209+0000 7fac2b9c7700 1 -- 172.21.15.52:0/3336929389 --> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fac2405f270 con 0x7fac10039f60 2023-12-16T20:45:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.209+0000 7fac19ffb700 1 -- 172.21.15.52:0/3336929389 <== mgr.14120 v2:172.21.15.52:6800/221719009 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fac2405f270 con 0x7fac10039f60 2023-12-16T20:45:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.209+0000 7fac2b9c7700 1 -- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fac10039f60 msgr2=0x7fac1003c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.209+0000 7fac2b9c7700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fac10039f60 0x7fac1003c340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fac140039d0 tx=0x7fac14009350 comp rx=0 tx=0).stop 2023-12-16T20:45:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.209+0000 7fac2b9c7700 1 -- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac24070f80 msgr2=0x7fac241029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.209+0000 7fac2b9c7700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac24070f80 0x7fac241029a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fac20006b00 tx=0x7fac20009250 comp rx=0 tx=0).stop 2023-12-16T20:45:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.209+0000 7fac2b9c7700 1 -- 172.21.15.52:0/3336929389 shutdown_connections 2023-12-16T20:45:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.209+0000 7fac2b9c7700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fac10039f60 0x7fac1003c340 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.209+0000 7fac2b9c7700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac24070f80 0x7fac241029a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.209+0000 7fac2b9c7700 1 --2- 172.21.15.52:0/3336929389 >> [v2:172.21.15.52:6800/2740978054,v1:172.21.15.52:6801/2740978054] conn(0x7fac100365f0 0x7fac10038aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.209+0000 7fac2b9c7700 1 -- 172.21.15.52:0/3336929389 >> 172.21.15.52:0/3336929389 conn(0x7fac240fc970 msgr2=0x7fac240fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.210+0000 7fac2b9c7700 1 -- 172.21.15.52:0/3336929389 shutdown_connections 2023-12-16T20:45:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.210+0000 7fac2b9c7700 1 -- 172.21.15.52:0/3336929389 wait complete. 2023-12-16T20:45:08.447 INFO:teuthology.orchestra.run.smithi052.stdout:mgr epoch 5 is available 2023-12-16T20:45:08.448 INFO:teuthology.orchestra.run.smithi052.stdout:Setting orchestrator backend to cephadm... 2023-12-16T20:45:08.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:08 smithi052 ceph-mon[80761]: Found migration_current of "None". Setting to last migration. 2023-12-16T20:45:08.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:08 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.nmkcbs/mirror_snapshot_schedule"}]: dispatch 2023-12-16T20:45:08.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:08 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.nmkcbs/trash_purge_schedule"}]: dispatch 2023-12-16T20:45:08.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:08 smithi052 ceph-mon[80761]: mgrmap e7: smithi052.nmkcbs(active, since 1.00536s) 2023-12-16T20:45:09.136 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.677+0000 7f896d09c700 1 Processor -- start 2023-12-16T20:45:09.136 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.678+0000 7f896d09c700 1 -- start start 2023-12-16T20:45:09.136 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.678+0000 7f896d09c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8968076c00 0x7f8968104f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:09.137 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.678+0000 7f896d09c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8968105440 con 0x7f8968076c00 2023-12-16T20:45:09.137 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.678+0000 7f8966d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8968076c00 0x7f8968104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:09.137 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.678+0000 7f8966d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8968076c00 0x7f8968104f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46396/0 (socket says 172.21.15.52:46396) 2023-12-16T20:45:09.137 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.678+0000 7f8966d9d700 1 -- 172.21.15.52:0/1213690279 learned_addr learned my addr 172.21.15.52:0/1213690279 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:09.137 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.678+0000 7f8966d9d700 1 -- 172.21.15.52:0/1213690279 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8968105580 con 0x7f8968076c00 2023-12-16T20:45:09.137 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.678+0000 7f8966d9d700 1 --2- 172.21.15.52:0/1213690279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8968076c00 0x7f8968104f00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f89580098e0 tx=0x7f8958009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4f7600cc5abe61c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:09.137 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.679+0000 7f8965d9b700 1 -- 172.21.15.52:0/1213690279 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f895800b860 con 0x7f8968076c00 2023-12-16T20:45:09.137 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.679+0000 7f8965d9b700 1 -- 172.21.15.52:0/1213690279 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f895800be30 con 0x7f8968076c00 2023-12-16T20:45:09.137 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.679+0000 7f8965d9b700 1 -- 172.21.15.52:0/1213690279 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8958010380 con 0x7f8968076c00 2023-12-16T20:45:09.137 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.679+0000 7f896d09c700 1 -- 172.21.15.52:0/1213690279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8968076c00 msgr2=0x7f8968104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:09.137 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.679+0000 7f896d09c700 1 --2- 172.21.15.52:0/1213690279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8968076c00 0x7f8968104f00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f89580098e0 tx=0x7f8958009250 comp rx=0 tx=0).stop 2023-12-16T20:45:09.137 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.679+0000 7f896d09c700 1 -- 172.21.15.52:0/1213690279 shutdown_connections 2023-12-16T20:45:09.137 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.679+0000 7f896d09c700 1 --2- 172.21.15.52:0/1213690279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8968076c00 0x7f8968104f00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.679+0000 7f896d09c700 1 -- 172.21.15.52:0/1213690279 >> 172.21.15.52:0/1213690279 conn(0x7f89680fc950 msgr2=0x7f89680fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.679+0000 7f896d09c700 1 -- 172.21.15.52:0/1213690279 shutdown_connections 2023-12-16T20:45:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.679+0000 7f896d09c700 1 -- 172.21.15.52:0/1213690279 wait complete. 2023-12-16T20:45:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.680+0000 7f896d09c700 1 Processor -- start 2023-12-16T20:45:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.680+0000 7f896d09c700 1 -- start start 2023-12-16T20:45:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.680+0000 7f896d09c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8968076c00 0x7f8968102980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.680+0000 7f896d09c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8968100e00 con 0x7f8968076c00 2023-12-16T20:45:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.680+0000 7f8966d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8968076c00 0x7f8968102980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.680+0000 7f8966d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8968076c00 0x7f8968102980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46402/0 (socket says 172.21.15.52:46402) 2023-12-16T20:45:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.680+0000 7f8966d9d700 1 -- 172.21.15.52:0/3763940869 learned_addr learned my addr 172.21.15.52:0/3763940869 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.681+0000 7f8966d9d700 1 -- 172.21.15.52:0/3763940869 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89580094f0 con 0x7f8968076c00 2023-12-16T20:45:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.681+0000 7f8966d9d700 1 --2- 172.21.15.52:0/3763940869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8968076c00 0x7f8968102980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8958006d40 tx=0x7f8958009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.681+0000 7f894f7fe700 1 -- 172.21.15.52:0/3763940869 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89580109b0 con 0x7f8968076c00 2023-12-16T20:45:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.681+0000 7f894f7fe700 1 -- 172.21.15.52:0/3763940869 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f8958024440 con 0x7f8968076c00 2023-12-16T20:45:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.681+0000 7f896d09c700 1 -- 172.21.15.52:0/3763940869 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8968101000 con 0x7f8968076c00 2023-12-16T20:45:09.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.681+0000 7f894f7fe700 1 -- 172.21.15.52:0/3763940869 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f895801a5f0 con 0x7f8968076c00 2023-12-16T20:45:09.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.681+0000 7f896d09c700 1 -- 172.21.15.52:0/3763940869 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8968101410 con 0x7f8968076c00 2023-12-16T20:45:09.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.682+0000 7f894f7fe700 1 -- 172.21.15.52:0/3763940869 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7f8958010b10 con 0x7f8968076c00 2023-12-16T20:45:09.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.682+0000 7f896d09c700 1 -- 172.21.15.52:0/3763940869 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896804bb80 con 0x7f8968076c00 2023-12-16T20:45:09.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.682+0000 7f894f7fe700 1 --2- 172.21.15.52:0/3763940869 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f8950036510 0x7f89500389c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:09.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.682+0000 7f894f7fe700 1 -- 172.21.15.52:0/3763940869 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f895804a230 con 0x7f8968076c00 2023-12-16T20:45:09.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.682+0000 7f896659c700 1 --2- 172.21.15.52:0/3763940869 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f8950036510 0x7f89500389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:09.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.683+0000 7f896659c700 1 --2- 172.21.15.52:0/3763940869 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f8950036510 0x7f89500389c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f895c009990 tx=0x7f895c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:09.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.687+0000 7f894f7fe700 1 -- 172.21.15.52:0/3763940869 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f89580245b0 con 0x7f8968076c00 2023-12-16T20:45:09.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.896+0000 7f896d09c700 1 -- 172.21.15.52:0/3763940869 --> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f896810adb0 con 0x7f8950036510 2023-12-16T20:45:09.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.903+0000 7f894f7fe700 1 -- 172.21.15.52:0/3763940869 <== mgr.14120 v2:172.21.15.52:6800/221719009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f896810adb0 con 0x7f8950036510 2023-12-16T20:45:09.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.905+0000 7f896d09c700 1 -- 172.21.15.52:0/3763940869 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f8950036510 msgr2=0x7f89500389c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:09.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.905+0000 7f896d09c700 1 --2- 172.21.15.52:0/3763940869 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f8950036510 0x7f89500389c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f895c009990 tx=0x7f895c006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:09.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.905+0000 7f896d09c700 1 -- 172.21.15.52:0/3763940869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8968076c00 msgr2=0x7f8968102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:09.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.905+0000 7f896d09c700 1 --2- 172.21.15.52:0/3763940869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8968076c00 0x7f8968102980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8958006d40 tx=0x7f8958009250 comp rx=0 tx=0).stop 2023-12-16T20:45:09.140 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.905+0000 7f896d09c700 1 -- 172.21.15.52:0/3763940869 shutdown_connections 2023-12-16T20:45:09.140 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.905+0000 7f896d09c700 1 --2- 172.21.15.52:0/3763940869 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f8950036510 0x7f89500389c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:09.140 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.905+0000 7f896d09c700 1 --2- 172.21.15.52:0/3763940869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8968076c00 0x7f8968102980 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:09.140 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.905+0000 7f896d09c700 1 -- 172.21.15.52:0/3763940869 >> 172.21.15.52:0/3763940869 conn(0x7f89680fc950 msgr2=0x7f89680fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:09.140 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.905+0000 7f896d09c700 1 -- 172.21.15.52:0/3763940869 shutdown_connections 2023-12-16T20:45:09.140 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:08.905+0000 7f896d09c700 1 -- 172.21.15.52:0/3763940869 wait complete. 2023-12-16T20:45:09.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:09 smithi052 ceph-mon[80761]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-16T20:45:09.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:09 smithi052 ceph-mon[80761]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-16T20:45:09.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:09 smithi052 ceph-mon[80761]: [16/Dec/2023:20:45:08] ENGINE Bus STARTING 2023-12-16T20:45:09.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:09 smithi052 ceph-mon[80761]: [16/Dec/2023:20:45:08] ENGINE Serving on https://172.21.15.52:7150 2023-12-16T20:45:09.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:09 smithi052 ceph-mon[80761]: [16/Dec/2023:20:45:08] ENGINE Bus STARTED 2023-12-16T20:45:09.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:09 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:09.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:09 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:45:09.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:09 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:09.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:09 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:45:09.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout value unchanged 2023-12-16T20:45:09.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.375+0000 7faa7a6b3700 1 Processor -- start 2023-12-16T20:45:09.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.375+0000 7faa7a6b3700 1 -- start start 2023-12-16T20:45:09.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.375+0000 7faa7a6b3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa741039c0 0x7faa74103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:09.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.375+0000 7faa7a6b3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa74104310 con 0x7faa741039c0 2023-12-16T20:45:09.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.375+0000 7faa73fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa741039c0 0x7faa74103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:09.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.375+0000 7faa73fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa741039c0 0x7faa74103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46414/0 (socket says 172.21.15.52:46414) 2023-12-16T20:45:09.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.375+0000 7faa73fff700 1 -- 172.21.15.52:0/387985255 learned_addr learned my addr 172.21.15.52:0/387985255 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:09.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.376+0000 7faa73fff700 1 -- 172.21.15.52:0/387985255 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa74101fb0 con 0x7faa741039c0 2023-12-16T20:45:09.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.376+0000 7faa73fff700 1 --2- 172.21.15.52:0/387985255 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa741039c0 0x7faa74103dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faa640098e0 tx=0x7faa64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950dafed16fd85f1 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:09.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.376+0000 7faa72ffd700 1 -- 172.21.15.52:0/387985255 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa6400b860 con 0x7faa741039c0 2023-12-16T20:45:09.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.376+0000 7faa72ffd700 1 -- 172.21.15.52:0/387985255 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7faa6400be30 con 0x7faa741039c0 2023-12-16T20:45:09.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.376+0000 7faa72ffd700 1 -- 172.21.15.52:0/387985255 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa64010380 con 0x7faa741039c0 2023-12-16T20:45:09.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.376+0000 7faa7a6b3700 1 -- 172.21.15.52:0/387985255 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa741039c0 msgr2=0x7faa74103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:09.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.376+0000 7faa7a6b3700 1 --2- 172.21.15.52:0/387985255 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa741039c0 0x7faa74103dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faa640098e0 tx=0x7faa64009250 comp rx=0 tx=0).stop 2023-12-16T20:45:09.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.377+0000 7faa7a6b3700 1 -- 172.21.15.52:0/387985255 shutdown_connections 2023-12-16T20:45:09.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.377+0000 7faa7a6b3700 1 --2- 172.21.15.52:0/387985255 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa741039c0 0x7faa74103dd0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:09.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.377+0000 7faa7a6b3700 1 -- 172.21.15.52:0/387985255 >> 172.21.15.52:0/387985255 conn(0x7faa740fda40 msgr2=0x7faa740ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:09.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.377+0000 7faa7a6b3700 1 -- 172.21.15.52:0/387985255 shutdown_connections 2023-12-16T20:45:09.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.377+0000 7faa7a6b3700 1 -- 172.21.15.52:0/387985255 wait complete. 2023-12-16T20:45:09.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.377+0000 7faa7a6b3700 1 Processor -- start 2023-12-16T20:45:09.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.377+0000 7faa7a6b3700 1 -- start start 2023-12-16T20:45:09.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.377+0000 7faa7a6b3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa741039c0 0x7faa741110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:09.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.377+0000 7faa7a6b3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa74111630 con 0x7faa741039c0 2023-12-16T20:45:09.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.378+0000 7faa73fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa741039c0 0x7faa741110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:09.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.378+0000 7faa73fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa741039c0 0x7faa741110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46422/0 (socket says 172.21.15.52:46422) 2023-12-16T20:45:09.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.378+0000 7faa73fff700 1 -- 172.21.15.52:0/290626794 learned_addr learned my addr 172.21.15.52:0/290626794 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:09.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.378+0000 7faa73fff700 1 -- 172.21.15.52:0/290626794 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa640094f0 con 0x7faa741039c0 2023-12-16T20:45:09.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.378+0000 7faa73fff700 1 --2- 172.21.15.52:0/290626794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa741039c0 0x7faa741110f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faa64000c00 tx=0x7faa64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:09.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.378+0000 7faa70ff9700 1 -- 172.21.15.52:0/290626794 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa64010a80 con 0x7faa741039c0 2023-12-16T20:45:09.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.378+0000 7faa70ff9700 1 -- 172.21.15.52:0/290626794 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7faa6401a430 con 0x7faa741039c0 2023-12-16T20:45:09.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.378+0000 7faa7a6b3700 1 -- 172.21.15.52:0/290626794 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa74111830 con 0x7faa741039c0 2023-12-16T20:45:09.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.378+0000 7faa70ff9700 1 -- 172.21.15.52:0/290626794 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa640245b0 con 0x7faa741039c0 2023-12-16T20:45:09.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.378+0000 7faa7a6b3700 1 -- 172.21.15.52:0/290626794 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa74111c40 con 0x7faa741039c0 2023-12-16T20:45:09.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.379+0000 7faa70ff9700 1 -- 172.21.15.52:0/290626794 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7faa64024820 con 0x7faa741039c0 2023-12-16T20:45:09.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.379+0000 7faa70ff9700 1 --2- 172.21.15.52:0/290626794 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7faa5c036510 0x7faa5c0389c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:09.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.379+0000 7faa7a6b3700 1 -- 172.21.15.52:0/290626794 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa74102d60 con 0x7faa741039c0 2023-12-16T20:45:09.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.379+0000 7faa70ff9700 1 -- 172.21.15.52:0/290626794 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7faa6404a5c0 con 0x7faa741039c0 2023-12-16T20:45:09.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.380+0000 7faa737fe700 1 --2- 172.21.15.52:0/290626794 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7faa5c036510 0x7faa5c0389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:09.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.380+0000 7faa737fe700 1 --2- 172.21.15.52:0/290626794 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7faa5c036510 0x7faa5c0389c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faa68009990 tx=0x7faa68006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:09.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.384+0000 7faa70ff9700 1 -- 172.21.15.52:0/290626794 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7faa640186d0 con 0x7faa741039c0 2023-12-16T20:45:09.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.582+0000 7faa7a6b3700 1 -- 172.21.15.52:0/290626794 --> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7faa7410adf0 con 0x7faa5c036510 2023-12-16T20:45:09.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.583+0000 7faa70ff9700 1 -- 172.21.15.52:0/290626794 <== mgr.14120 v2:172.21.15.52:6800/221719009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7faa7410adf0 con 0x7faa5c036510 2023-12-16T20:45:09.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.585+0000 7faa7a6b3700 1 -- 172.21.15.52:0/290626794 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7faa5c036510 msgr2=0x7faa5c0389c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:09.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.585+0000 7faa7a6b3700 1 --2- 172.21.15.52:0/290626794 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7faa5c036510 0x7faa5c0389c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faa68009990 tx=0x7faa68006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:09.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.585+0000 7faa7a6b3700 1 -- 172.21.15.52:0/290626794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa741039c0 msgr2=0x7faa741110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:09.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.585+0000 7faa7a6b3700 1 --2- 172.21.15.52:0/290626794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa741039c0 0x7faa741110f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faa64000c00 tx=0x7faa64009250 comp rx=0 tx=0).stop 2023-12-16T20:45:09.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.585+0000 7faa7a6b3700 1 -- 172.21.15.52:0/290626794 shutdown_connections 2023-12-16T20:45:09.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.585+0000 7faa7a6b3700 1 --2- 172.21.15.52:0/290626794 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7faa5c036510 0x7faa5c0389c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:09.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.585+0000 7faa7a6b3700 1 --2- 172.21.15.52:0/290626794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa741039c0 0x7faa741110f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:09.850 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.585+0000 7faa7a6b3700 1 -- 172.21.15.52:0/290626794 >> 172.21.15.52:0/290626794 conn(0x7faa740fda40 msgr2=0x7faa740ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:09.850 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.585+0000 7faa7a6b3700 1 -- 172.21.15.52:0/290626794 shutdown_connections 2023-12-16T20:45:09.850 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:09.585+0000 7faa7a6b3700 1 -- 172.21.15.52:0/290626794 wait complete. 2023-12-16T20:45:09.850 INFO:teuthology.orchestra.run.smithi052.stdout:Generating ssh key... 2023-12-16T20:45:10.586 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:10 smithi052 ceph-mon[80761]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:10.586 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:10 smithi052 ceph-mon[80761]: mgrmap e8: smithi052.nmkcbs(active, since 2s) 2023-12-16T20:45:10.595 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.066+0000 7f76c0300700 1 Processor -- start 2023-12-16T20:45:10.595 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.066+0000 7f76c0300700 1 -- start start 2023-12-16T20:45:10.595 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.066+0000 7f76c0300700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76b81022d0 0x7f76b81026e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:10.595 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.066+0000 7f76c0300700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b8102c20 con 0x7f76b81022d0 2023-12-16T20:45:10.595 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.067+0000 7f76be09c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76b81022d0 0x7f76b81026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:10.595 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.067+0000 7f76be09c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76b81022d0 0x7f76b81026e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46432/0 (socket says 172.21.15.52:46432) 2023-12-16T20:45:10.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.067+0000 7f76be09c700 1 -- 172.21.15.52:0/2159751647 learned_addr learned my addr 172.21.15.52:0/2159751647 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:10.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.067+0000 7f76be09c700 1 -- 172.21.15.52:0/2159751647 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b8102d60 con 0x7f76b81022d0 2023-12-16T20:45:10.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.067+0000 7f76be09c700 1 --2- 172.21.15.52:0/2159751647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76b81022d0 0x7f76b81026e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f76b0009e50 tx=0x7f76b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41413b8d3e82462 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:10.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.067+0000 7f76bd09a700 1 -- 172.21.15.52:0/2159751647 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76b00092e0 con 0x7f76b81022d0 2023-12-16T20:45:10.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.067+0000 7f76bd09a700 1 -- 172.21.15.52:0/2159751647 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f76b000be90 con 0x7f76b81022d0 2023-12-16T20:45:10.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.067+0000 7f76bd09a700 1 -- 172.21.15.52:0/2159751647 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76b0007400 con 0x7f76b81022d0 2023-12-16T20:45:10.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.068+0000 7f76c0300700 1 -- 172.21.15.52:0/2159751647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76b81022d0 msgr2=0x7f76b81026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:10.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.068+0000 7f76c0300700 1 --2- 172.21.15.52:0/2159751647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76b81022d0 0x7f76b81026e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f76b0009e50 tx=0x7f76b0009840 comp rx=0 tx=0).stop 2023-12-16T20:45:10.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.068+0000 7f76c0300700 1 -- 172.21.15.52:0/2159751647 shutdown_connections 2023-12-16T20:45:10.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.068+0000 7f76c0300700 1 --2- 172.21.15.52:0/2159751647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76b81022d0 0x7f76b81026e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:10.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.068+0000 7f76c0300700 1 -- 172.21.15.52:0/2159751647 >> 172.21.15.52:0/2159751647 conn(0x7f76b80fda50 msgr2=0x7f76b80ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:10.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.068+0000 7f76c0300700 1 -- 172.21.15.52:0/2159751647 shutdown_connections 2023-12-16T20:45:10.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.068+0000 7f76c0300700 1 -- 172.21.15.52:0/2159751647 wait complete. 2023-12-16T20:45:10.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.068+0000 7f76c0300700 1 Processor -- start 2023-12-16T20:45:10.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.068+0000 7f76c0300700 1 -- start start 2023-12-16T20:45:10.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.069+0000 7f76c0300700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76b81022d0 0x7f76b8077c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:10.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.069+0000 7f76c0300700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b8076080 con 0x7f76b81022d0 2023-12-16T20:45:10.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.069+0000 7f76be09c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76b81022d0 0x7f76b8077c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:10.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.069+0000 7f76be09c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76b81022d0 0x7f76b8077c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46444/0 (socket says 172.21.15.52:46444) 2023-12-16T20:45:10.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.069+0000 7f76be09c700 1 -- 172.21.15.52:0/3776583728 learned_addr learned my addr 172.21.15.52:0/3776583728 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:10.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.069+0000 7f76be09c700 1 -- 172.21.15.52:0/3776583728 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b0009ae0 con 0x7f76b81022d0 2023-12-16T20:45:10.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.069+0000 7f76be09c700 1 --2- 172.21.15.52:0/3776583728 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76b81022d0 0x7f76b8077c00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f76b000b740 tx=0x7f76b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:10.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.069+0000 7f76b6ffd700 1 -- 172.21.15.52:0/3776583728 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76b0007670 con 0x7f76b81022d0 2023-12-16T20:45:10.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.069+0000 7f76c0300700 1 -- 172.21.15.52:0/3776583728 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b8076280 con 0x7f76b81022d0 2023-12-16T20:45:10.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.069+0000 7f76b6ffd700 1 -- 172.21.15.52:0/3776583728 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f76b0007c40 con 0x7f76b81022d0 2023-12-16T20:45:10.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.069+0000 7f76b6ffd700 1 -- 172.21.15.52:0/3776583728 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76b001cb10 con 0x7f76b81022d0 2023-12-16T20:45:10.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.069+0000 7f76c0300700 1 -- 172.21.15.52:0/3776583728 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b80766f0 con 0x7f76b81022d0 2023-12-16T20:45:10.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.070+0000 7f76b6ffd700 1 -- 172.21.15.52:0/3776583728 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f76b0019070 con 0x7f76b81022d0 2023-12-16T20:45:10.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.070+0000 7f76c0300700 1 -- 172.21.15.52:0/3776583728 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b810b150 con 0x7f76b81022d0 2023-12-16T20:45:10.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.070+0000 7f76b6ffd700 1 --2- 172.21.15.52:0/3776583728 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f76a00365f0 0x7f76a0038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:10.598 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.070+0000 7f76b6ffd700 1 -- 172.21.15.52:0/3776583728 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f76b0049eb0 con 0x7f76b81022d0 2023-12-16T20:45:10.598 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.071+0000 7f76bd89b700 1 --2- 172.21.15.52:0/3776583728 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f76a00365f0 0x7f76a0038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:10.598 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.071+0000 7f76bd89b700 1 --2- 172.21.15.52:0/3776583728 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f76a00365f0 0x7f76a0038aa0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f76ac009990 tx=0x7f76ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:10.598 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.075+0000 7f76b6ffd700 1 -- 172.21.15.52:0/3776583728 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f76b0019b10 con 0x7f76b81022d0 2023-12-16T20:45:10.598 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.271+0000 7f76c0300700 1 -- 172.21.15.52:0/3776583728 --> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f76b80769f0 con 0x7f76a00365f0 2023-12-16T20:45:10.598 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.368+0000 7f76b6ffd700 1 -- 172.21.15.52:0/3776583728 <== mgr.14120 v2:172.21.15.52:6800/221719009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f76b80769f0 con 0x7f76a00365f0 2023-12-16T20:45:10.598 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.371+0000 7f76c0300700 1 -- 172.21.15.52:0/3776583728 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f76a00365f0 msgr2=0x7f76a0038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:10.598 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.371+0000 7f76c0300700 1 --2- 172.21.15.52:0/3776583728 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f76a00365f0 0x7f76a0038aa0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f76ac009990 tx=0x7f76ac006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:10.598 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.371+0000 7f76c0300700 1 -- 172.21.15.52:0/3776583728 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76b81022d0 msgr2=0x7f76b8077c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:10.598 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.371+0000 7f76c0300700 1 --2- 172.21.15.52:0/3776583728 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76b81022d0 0x7f76b8077c00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f76b000b740 tx=0x7f76b0009840 comp rx=0 tx=0).stop 2023-12-16T20:45:10.598 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.371+0000 7f76c0300700 1 -- 172.21.15.52:0/3776583728 shutdown_connections 2023-12-16T20:45:10.598 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.371+0000 7f76c0300700 1 --2- 172.21.15.52:0/3776583728 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f76a00365f0 0x7f76a0038aa0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:10.598 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.371+0000 7f76c0300700 1 --2- 172.21.15.52:0/3776583728 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76b81022d0 0x7f76b8077c00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:10.598 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.371+0000 7f76c0300700 1 -- 172.21.15.52:0/3776583728 >> 172.21.15.52:0/3776583728 conn(0x7f76b80fda50 msgr2=0x7f76b80ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:10.598 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.371+0000 7f76c0300700 1 -- 172.21.15.52:0/3776583728 shutdown_connections 2023-12-16T20:45:10.599 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.371+0000 7f76c0300700 1 -- 172.21.15.52:0/3776583728 wait complete. 2023-12-16T20:45:11.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCezOur/AmELSSiiE6a38SQZw7Jntk9AxQ+ZI6ZlR1kzYJkaGTeHs35BiHZ53juUDGE1L1h9DloM08/zXDCKjtahqRMoKdmSJXfEGYsqKSGBzI9xxXSIQQfSPypd61ZjGXZsvtdTqZOh5r/yMQl7EmiCnnLbgDv5HIERhvn4cMo2ojp/nlaozPyManYEJHmbeUsJuwlArdEHa1eSGTD4CHffllirz13JKTS/nJJX4MHdV9uF377fLKmn3rf5A2dmkSp8onFTGzFxTxahnWS9Z1AEzTjcPRpz9+hJAjEbUjP4Q/JdEPOJKQvOyg+Oj+qR18E1L3lZQ28e4GT/Df4sUngWkHAzkjcJmY8VeGoTWfq/sV1SbGawxueIDEftsvjRxzc3oZDl1Qf1QpOSTMRHdothp3JuG9y7s2Kn5XA+u6G8uKRE1g0/8KaGMjQvQH0KU2mPe8Wxu/YxTuD0oVPJr22GSby8bPhYijl43LkvsqJdvn9XE3xccOQ2BMyjQIS/Uc= ceph-cf043a8a-9c53-11ee-95a5-87774f69a715 2023-12-16T20:45:11.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.815+0000 7f62f81a0700 1 Processor -- start 2023-12-16T20:45:11.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.815+0000 7f62f81a0700 1 -- start start 2023-12-16T20:45:11.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.815+0000 7f62f81a0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f01039b0 0x7f62f0103dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:11.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.815+0000 7f62f81a0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f0104300 con 0x7f62f01039b0 2023-12-16T20:45:11.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.815+0000 7f62f5f3c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f01039b0 0x7f62f0103dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:11.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.815+0000 7f62f5f3c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f01039b0 0x7f62f0103dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46452/0 (socket says 172.21.15.52:46452) 2023-12-16T20:45:11.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.815+0000 7f62f5f3c700 1 -- 172.21.15.52:0/4072558906 learned_addr learned my addr 172.21.15.52:0/4072558906 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:11.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.815+0000 7f62f5f3c700 1 -- 172.21.15.52:0/4072558906 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f0104440 con 0x7f62f01039b0 2023-12-16T20:45:11.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.816+0000 7f62f5f3c700 1 --2- 172.21.15.52:0/4072558906 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f01039b0 0x7f62f0103dc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0098e0 tx=0x7f62ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61dffa31e7c960b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:11.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.816+0000 7f62f4f3a700 1 -- 172.21.15.52:0/4072558906 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ec00b860 con 0x7f62f01039b0 2023-12-16T20:45:11.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.816+0000 7f62f4f3a700 1 -- 172.21.15.52:0/4072558906 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f62ec00be30 con 0x7f62f01039b0 2023-12-16T20:45:11.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.816+0000 7f62f4f3a700 1 -- 172.21.15.52:0/4072558906 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ec010380 con 0x7f62f01039b0 2023-12-16T20:45:11.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.816+0000 7f62f81a0700 1 -- 172.21.15.52:0/4072558906 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f01039b0 msgr2=0x7f62f0103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:11.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.816+0000 7f62f81a0700 1 --2- 172.21.15.52:0/4072558906 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f01039b0 0x7f62f0103dc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0098e0 tx=0x7f62ec009250 comp rx=0 tx=0).stop 2023-12-16T20:45:11.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.816+0000 7f62f81a0700 1 -- 172.21.15.52:0/4072558906 shutdown_connections 2023-12-16T20:45:11.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.816+0000 7f62f81a0700 1 --2- 172.21.15.52:0/4072558906 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f01039b0 0x7f62f0103dc0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:11.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.816+0000 7f62f81a0700 1 -- 172.21.15.52:0/4072558906 >> 172.21.15.52:0/4072558906 conn(0x7f62f00fc0b0 msgr2=0x7f62f00fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:11.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.817+0000 7f62f81a0700 1 -- 172.21.15.52:0/4072558906 shutdown_connections 2023-12-16T20:45:11.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.817+0000 7f62f81a0700 1 -- 172.21.15.52:0/4072558906 wait complete. 2023-12-16T20:45:11.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.817+0000 7f62f81a0700 1 Processor -- start 2023-12-16T20:45:11.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.817+0000 7f62f81a0700 1 -- start start 2023-12-16T20:45:11.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.817+0000 7f62f81a0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f01039b0 0x7f62f0111130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:11.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.817+0000 7f62f81a0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f0111670 con 0x7f62f01039b0 2023-12-16T20:45:11.239 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.817+0000 7f62f5f3c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f01039b0 0x7f62f0111130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:11.239 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.817+0000 7f62f5f3c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f01039b0 0x7f62f0111130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46460/0 (socket says 172.21.15.52:46460) 2023-12-16T20:45:11.239 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.817+0000 7f62f5f3c700 1 -- 172.21.15.52:0/31618476 learned_addr learned my addr 172.21.15.52:0/31618476 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:11.239 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.818+0000 7f62f5f3c700 1 -- 172.21.15.52:0/31618476 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec0094f0 con 0x7f62f01039b0 2023-12-16T20:45:11.239 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.818+0000 7f62f5f3c700 1 --2- 172.21.15.52:0/31618476 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f01039b0 0x7f62f0111130 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f62ec006b00 tx=0x7f62ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:11.239 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.818+0000 7f62e67fc700 1 -- 172.21.15.52:0/31618476 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ec0109b0 con 0x7f62f01039b0 2023-12-16T20:45:11.239 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.818+0000 7f62e67fc700 1 -- 172.21.15.52:0/31618476 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f62ec024440 con 0x7f62f01039b0 2023-12-16T20:45:11.239 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.818+0000 7f62f81a0700 1 -- 172.21.15.52:0/31618476 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f0111870 con 0x7f62f01039b0 2023-12-16T20:45:11.239 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.818+0000 7f62e67fc700 1 -- 172.21.15.52:0/31618476 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ec01a5f0 con 0x7f62f01039b0 2023-12-16T20:45:11.239 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.818+0000 7f62f81a0700 1 -- 172.21.15.52:0/31618476 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f0111c80 con 0x7f62f01039b0 2023-12-16T20:45:11.239 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.819+0000 7f62e67fc700 1 -- 172.21.15.52:0/31618476 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f62ec01a8b0 con 0x7f62f01039b0 2023-12-16T20:45:11.239 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.819+0000 7f62f81a0700 1 -- 172.21.15.52:0/31618476 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f004bb80 con 0x7f62f01039b0 2023-12-16T20:45:11.239 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.819+0000 7f62e67fc700 1 --2- 172.21.15.52:0/31618476 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f62dc0365f0 0x7f62dc038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:11.239 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.819+0000 7f62e67fc700 1 -- 172.21.15.52:0/31618476 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f62ec04a720 con 0x7f62f01039b0 2023-12-16T20:45:11.240 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.819+0000 7f62f573b700 1 --2- 172.21.15.52:0/31618476 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f62dc0365f0 0x7f62dc038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:11.240 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.820+0000 7f62f573b700 1 --2- 172.21.15.52:0/31618476 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f62dc0365f0 0x7f62dc038aa0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009990 tx=0x7f62e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:11.240 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:10.824+0000 7f62e67fc700 1 -- 172.21.15.52:0/31618476 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f62ec018530 con 0x7f62f01039b0 2023-12-16T20:45:11.240 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.021+0000 7f62f81a0700 1 -- 172.21.15.52:0/31618476 --> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f62f010b000 con 0x7f62dc0365f0 2023-12-16T20:45:11.240 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.022+0000 7f62e67fc700 1 -- 172.21.15.52:0/31618476 <== mgr.14120 v2:172.21.15.52:6800/221719009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f62f010b000 con 0x7f62dc0365f0 2023-12-16T20:45:11.240 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.024+0000 7f62f81a0700 1 -- 172.21.15.52:0/31618476 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f62dc0365f0 msgr2=0x7f62dc038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:11.240 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.024+0000 7f62f81a0700 1 --2- 172.21.15.52:0/31618476 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f62dc0365f0 0x7f62dc038aa0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009990 tx=0x7f62e0006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:11.240 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.024+0000 7f62f81a0700 1 -- 172.21.15.52:0/31618476 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f01039b0 msgr2=0x7f62f0111130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:11.240 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.024+0000 7f62f81a0700 1 --2- 172.21.15.52:0/31618476 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f01039b0 0x7f62f0111130 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f62ec006b00 tx=0x7f62ec009250 comp rx=0 tx=0).stop 2023-12-16T20:45:11.240 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.024+0000 7f62f81a0700 1 -- 172.21.15.52:0/31618476 shutdown_connections 2023-12-16T20:45:11.240 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.024+0000 7f62f81a0700 1 --2- 172.21.15.52:0/31618476 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f62dc0365f0 0x7f62dc038aa0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:11.241 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.024+0000 7f62f81a0700 1 --2- 172.21.15.52:0/31618476 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f01039b0 0x7f62f0111130 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:11.241 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.024+0000 7f62f81a0700 1 -- 172.21.15.52:0/31618476 >> 172.21.15.52:0/31618476 conn(0x7f62f00fc0b0 msgr2=0x7f62f00fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:11.241 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.024+0000 7f62f81a0700 1 -- 172.21.15.52:0/31618476 shutdown_connections 2023-12-16T20:45:11.241 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.024+0000 7f62f81a0700 1 -- 172.21.15.52:0/31618476 wait complete. 2023-12-16T20:45:11.241 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-12-16T20:45:11.241 INFO:teuthology.orchestra.run.smithi052.stdout:Adding key to root@localhost authorized_keys... 2023-12-16T20:45:11.241 INFO:teuthology.orchestra.run.smithi052.stdout:Adding host smithi052... 2023-12-16T20:45:11.672 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:11 smithi052 ceph-mon[80761]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:11.672 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:11 smithi052 ceph-mon[80761]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:11.672 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:11 smithi052 ceph-mon[80761]: Generating ssh key... 2023-12-16T20:45:11.672 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:11 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:11.672 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:11 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:12.674 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:12 smithi052 ceph-mon[80761]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:13.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:13 smithi052 ceph-mon[80761]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi052", "addr": "172.21.15.52", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:13.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:13 smithi052 ceph-mon[80761]: Deploying cephadm binary to smithi052 2023-12-16T20:45:13.983 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Added host 'smithi052' with addr '172.21.15.52' 2023-12-16T20:45:14.035 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.460+0000 7fa96fc99700 1 Processor -- start 2023-12-16T20:45:14.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.461+0000 7fa96fc99700 1 -- start start 2023-12-16T20:45:14.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.461+0000 7fa96fc99700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9681011d0 0x7fa9681015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:14.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.461+0000 7fa96fc99700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa968101b20 con 0x7fa9681011d0 2023-12-16T20:45:14.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.461+0000 7fa96da35700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9681011d0 0x7fa9681015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:14.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.461+0000 7fa96da35700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9681011d0 0x7fa9681015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46474/0 (socket says 172.21.15.52:46474) 2023-12-16T20:45:14.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.461+0000 7fa96da35700 1 -- 172.21.15.52:0/1215220880 learned_addr learned my addr 172.21.15.52:0/1215220880 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:14.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.461+0000 7fa96da35700 1 -- 172.21.15.52:0/1215220880 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa968101c60 con 0x7fa9681011d0 2023-12-16T20:45:14.036 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.462+0000 7fa96da35700 1 --2- 172.21.15.52:0/1215220880 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9681011d0 0x7fa9681015e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa9640098e0 tx=0x7fa964009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca2092511866c4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:14.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.462+0000 7fa96ca33700 1 -- 172.21.15.52:0/1215220880 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa96400b860 con 0x7fa9681011d0 2023-12-16T20:45:14.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.462+0000 7fa96ca33700 1 -- 172.21.15.52:0/1215220880 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa96400be30 con 0x7fa9681011d0 2023-12-16T20:45:14.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.462+0000 7fa96ca33700 1 -- 172.21.15.52:0/1215220880 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa964010380 con 0x7fa9681011d0 2023-12-16T20:45:14.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.462+0000 7fa96fc99700 1 -- 172.21.15.52:0/1215220880 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9681011d0 msgr2=0x7fa9681015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:14.046 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.462+0000 7fa96fc99700 1 --2- 172.21.15.52:0/1215220880 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9681011d0 0x7fa9681015e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa9640098e0 tx=0x7fa964009250 comp rx=0 tx=0).stop 2023-12-16T20:45:14.046 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.462+0000 7fa96fc99700 1 -- 172.21.15.52:0/1215220880 shutdown_connections 2023-12-16T20:45:14.046 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.462+0000 7fa96fc99700 1 --2- 172.21.15.52:0/1215220880 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9681011d0 0x7fa9681015e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:14.046 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.462+0000 7fa96fc99700 1 -- 172.21.15.52:0/1215220880 >> 172.21.15.52:0/1215220880 conn(0x7fa9680fc950 msgr2=0x7fa9680fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:14.046 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.462+0000 7fa96fc99700 1 -- 172.21.15.52:0/1215220880 shutdown_connections 2023-12-16T20:45:14.047 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.462+0000 7fa96fc99700 1 -- 172.21.15.52:0/1215220880 wait complete. 2023-12-16T20:45:14.047 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.463+0000 7fa96fc99700 1 Processor -- start 2023-12-16T20:45:14.047 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.463+0000 7fa96fc99700 1 -- start start 2023-12-16T20:45:14.047 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.463+0000 7fa96fc99700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9681011d0 0x7fa968111150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:14.047 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.463+0000 7fa96fc99700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa968111690 con 0x7fa9681011d0 2023-12-16T20:45:14.047 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.463+0000 7fa96da35700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9681011d0 0x7fa968111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:14.047 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.463+0000 7fa96da35700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9681011d0 0x7fa968111150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46482/0 (socket says 172.21.15.52:46482) 2023-12-16T20:45:14.047 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.463+0000 7fa96da35700 1 -- 172.21.15.52:0/2130426841 learned_addr learned my addr 172.21.15.52:0/2130426841 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:14.047 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.464+0000 7fa96da35700 1 -- 172.21.15.52:0/2130426841 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9640094f0 con 0x7fa9681011d0 2023-12-16T20:45:14.047 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.464+0000 7fa96da35700 1 --2- 172.21.15.52:0/2130426841 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9681011d0 0x7fa968111150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa964000c00 tx=0x7fa964009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:14.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.464+0000 7fa95e7fc700 1 -- 172.21.15.52:0/2130426841 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa964010a80 con 0x7fa9681011d0 2023-12-16T20:45:14.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.464+0000 7fa95e7fc700 1 -- 172.21.15.52:0/2130426841 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa96401a430 con 0x7fa9681011d0 2023-12-16T20:45:14.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.464+0000 7fa96fc99700 1 -- 172.21.15.52:0/2130426841 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa968111890 con 0x7fa9681011d0 2023-12-16T20:45:14.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.464+0000 7fa95e7fc700 1 -- 172.21.15.52:0/2130426841 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9640245b0 con 0x7fa9681011d0 2023-12-16T20:45:14.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.464+0000 7fa96fc99700 1 -- 172.21.15.52:0/2130426841 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa968111ca0 con 0x7fa9681011d0 2023-12-16T20:45:14.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.465+0000 7fa95e7fc700 1 -- 172.21.15.52:0/2130426841 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fa96401a5a0 con 0x7fa9681011d0 2023-12-16T20:45:14.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.465+0000 7fa95e7fc700 1 --2- 172.21.15.52:0/2130426841 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fa954036640 0x7fa954038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:14.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.465+0000 7fa96fc99700 1 -- 172.21.15.52:0/2130426841 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa96804bb80 con 0x7fa9681011d0 2023-12-16T20:45:14.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.465+0000 7fa95e7fc700 1 -- 172.21.15.52:0/2130426841 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa96404ab20 con 0x7fa9681011d0 2023-12-16T20:45:14.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.465+0000 7fa96d234700 1 --2- 172.21.15.52:0/2130426841 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fa954036640 0x7fa954038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:14.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.466+0000 7fa96d234700 1 --2- 172.21.15.52:0/2130426841 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fa954036640 0x7fa954038af0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa958009990 tx=0x7fa958006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:14.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.470+0000 7fa95e7fc700 1 -- 172.21.15.52:0/2130426841 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa964023df0 con 0x7fa9681011d0 2023-12-16T20:45:14.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:11.670+0000 7fa96fc99700 1 -- 172.21.15.52:0/2130426841 --> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi052", "addr": "172.21.15.52", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9680febc0 con 0x7fa954036640 2023-12-16T20:45:14.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:13.750+0000 7fa95e7fc700 1 -- 172.21.15.52:0/2130426841 <== mgr.14120 v2:172.21.15.52:6800/221719009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fa9680febc0 con 0x7fa954036640 2023-12-16T20:45:14.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:13.753+0000 7fa96fc99700 1 -- 172.21.15.52:0/2130426841 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fa954036640 msgr2=0x7fa954038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:14.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:13.753+0000 7fa96fc99700 1 --2- 172.21.15.52:0/2130426841 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fa954036640 0x7fa954038af0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa958009990 tx=0x7fa958006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:14.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:13.753+0000 7fa96fc99700 1 -- 172.21.15.52:0/2130426841 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9681011d0 msgr2=0x7fa968111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:14.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:13.753+0000 7fa96fc99700 1 --2- 172.21.15.52:0/2130426841 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9681011d0 0x7fa968111150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa964000c00 tx=0x7fa964009250 comp rx=0 tx=0).stop 2023-12-16T20:45:14.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:13.753+0000 7fa96fc99700 1 -- 172.21.15.52:0/2130426841 shutdown_connections 2023-12-16T20:45:14.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:13.753+0000 7fa96fc99700 1 --2- 172.21.15.52:0/2130426841 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fa954036640 0x7fa954038af0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:14.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:13.753+0000 7fa96fc99700 1 --2- 172.21.15.52:0/2130426841 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9681011d0 0x7fa968111150 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:14.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:13.753+0000 7fa96fc99700 1 -- 172.21.15.52:0/2130426841 >> 172.21.15.52:0/2130426841 conn(0x7fa9680fc950 msgr2=0x7fa9680767f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:14.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:13.753+0000 7fa96fc99700 1 -- 172.21.15.52:0/2130426841 shutdown_connections 2023-12-16T20:45:14.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:13.753+0000 7fa96fc99700 1 -- 172.21.15.52:0/2130426841 wait complete. 2023-12-16T20:45:14.050 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying mon service with default placement... 2023-12-16T20:45:14.753 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-12-16T20:45:14.753 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.238+0000 7f6ce4da6700 1 Processor -- start 2023-12-16T20:45:14.753 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.238+0000 7f6ce4da6700 1 -- start start 2023-12-16T20:45:14.753 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.238+0000 7f6ce4da6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ce01039c0 0x7f6ce0103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:14.753 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.238+0000 7f6ce4da6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce0104310 con 0x7f6ce01039c0 2023-12-16T20:45:14.753 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.238+0000 7f6cde59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ce01039c0 0x7f6ce0103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:14.753 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.238+0000 7f6cde59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ce01039c0 0x7f6ce0103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45160/0 (socket says 172.21.15.52:45160) 2023-12-16T20:45:14.753 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.238+0000 7f6cde59c700 1 -- 172.21.15.52:0/3462812986 learned_addr learned my addr 172.21.15.52:0/3462812986 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:14.753 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.238+0000 7f6cde59c700 1 -- 172.21.15.52:0/3462812986 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce0101fb0 con 0x7f6ce01039c0 2023-12-16T20:45:14.753 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.239+0000 7f6cde59c700 1 --2- 172.21.15.52:0/3462812986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ce01039c0 0x7f6ce0103dd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00098e0 tx=0x7f6cd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33cc468d812785d5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.239+0000 7f6cdd59a700 1 -- 172.21.15.52:0/3462812986 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cd000b860 con 0x7f6ce01039c0 2023-12-16T20:45:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.239+0000 7f6cdd59a700 1 -- 172.21.15.52:0/3462812986 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f6cd000be30 con 0x7f6ce01039c0 2023-12-16T20:45:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.239+0000 7f6cdd59a700 1 -- 172.21.15.52:0/3462812986 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cd0010380 con 0x7f6ce01039c0 2023-12-16T20:45:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.239+0000 7f6ce4da6700 1 -- 172.21.15.52:0/3462812986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ce01039c0 msgr2=0x7f6ce0103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.239+0000 7f6ce4da6700 1 --2- 172.21.15.52:0/3462812986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ce01039c0 0x7f6ce0103dd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00098e0 tx=0x7f6cd0009250 comp rx=0 tx=0).stop 2023-12-16T20:45:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.240+0000 7f6ce4da6700 1 -- 172.21.15.52:0/3462812986 shutdown_connections 2023-12-16T20:45:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.240+0000 7f6ce4da6700 1 --2- 172.21.15.52:0/3462812986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ce01039c0 0x7f6ce0103dd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.240+0000 7f6ce4da6700 1 -- 172.21.15.52:0/3462812986 >> 172.21.15.52:0/3462812986 conn(0x7f6ce00fda40 msgr2=0x7f6ce00ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.240+0000 7f6ce4da6700 1 -- 172.21.15.52:0/3462812986 shutdown_connections 2023-12-16T20:45:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.240+0000 7f6ce4da6700 1 -- 172.21.15.52:0/3462812986 wait complete. 2023-12-16T20:45:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.240+0000 7f6ce4da6700 1 Processor -- start 2023-12-16T20:45:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.240+0000 7f6ce4da6700 1 -- start start 2023-12-16T20:45:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.240+0000 7f6ce4da6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ce01039c0 0x7f6ce0078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.240+0000 7f6ce4da6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce0077190 con 0x7f6ce01039c0 2023-12-16T20:45:14.754 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.240+0000 7f6cde59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ce01039c0 0x7f6ce0078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:14.755 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.240+0000 7f6cde59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ce01039c0 0x7f6ce0078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45168/0 (socket says 172.21.15.52:45168) 2023-12-16T20:45:14.755 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.240+0000 7f6cde59c700 1 -- 172.21.15.52:0/522352797 learned_addr learned my addr 172.21.15.52:0/522352797 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:14.755 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.241+0000 7f6cde59c700 1 -- 172.21.15.52:0/522352797 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd00094f0 con 0x7f6ce01039c0 2023-12-16T20:45:14.755 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.241+0000 7f6cde59c700 1 --2- 172.21.15.52:0/522352797 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ce01039c0 0x7f6ce0078d10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0000c00 tx=0x7f6cd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:14.755 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.241+0000 7f6cceffd700 1 -- 172.21.15.52:0/522352797 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cd0010a80 con 0x7f6ce01039c0 2023-12-16T20:45:14.755 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.241+0000 7f6cceffd700 1 -- 172.21.15.52:0/522352797 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f6cd001a430 con 0x7f6ce01039c0 2023-12-16T20:45:14.755 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.241+0000 7f6ce4da6700 1 -- 172.21.15.52:0/522352797 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce0077390 con 0x7f6ce01039c0 2023-12-16T20:45:14.755 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.241+0000 7f6ce4da6700 1 -- 172.21.15.52:0/522352797 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce00777a0 con 0x7f6ce01039c0 2023-12-16T20:45:14.755 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.242+0000 7f6cceffd700 1 -- 172.21.15.52:0/522352797 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cd00245b0 con 0x7f6ce01039c0 2023-12-16T20:45:14.755 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.242+0000 7f6cceffd700 1 -- 172.21.15.52:0/522352797 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f6cd0024820 con 0x7f6ce01039c0 2023-12-16T20:45:14.755 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.242+0000 7f6cceffd700 1 --2- 172.21.15.52:0/522352797 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f6cc80362c0 0x7f6cc8038770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:14.756 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.242+0000 7f6ce4da6700 1 -- 172.21.15.52:0/522352797 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce005f270 con 0x7f6ce01039c0 2023-12-16T20:45:14.756 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.242+0000 7f6cceffd700 1 -- 172.21.15.52:0/522352797 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6cd004a6a0 con 0x7f6ce01039c0 2023-12-16T20:45:14.756 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.242+0000 7f6cddd9b700 1 --2- 172.21.15.52:0/522352797 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f6cc80362c0 0x7f6cc8038770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:14.756 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.243+0000 7f6cddd9b700 1 --2- 172.21.15.52:0/522352797 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f6cc80362c0 0x7f6cc8038770 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4009990 tx=0x7f6cd4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:14.756 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.247+0000 7f6cceffd700 1 -- 172.21.15.52:0/522352797 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6cd002c340 con 0x7f6ce01039c0 2023-12-16T20:45:14.756 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.458+0000 7f6ce4da6700 1 -- 172.21.15.52:0/522352797 --> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ce010adc0 con 0x7f6cc80362c0 2023-12-16T20:45:14.756 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.462+0000 7f6cceffd700 1 -- 172.21.15.52:0/522352797 <== mgr.14120 v2:172.21.15.52:6800/221719009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6ce010adc0 con 0x7f6cc80362c0 2023-12-16T20:45:14.756 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.464+0000 7f6ce4da6700 1 -- 172.21.15.52:0/522352797 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f6cc80362c0 msgr2=0x7f6cc8038770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:14.756 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.464+0000 7f6ce4da6700 1 --2- 172.21.15.52:0/522352797 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f6cc80362c0 0x7f6cc8038770 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4009990 tx=0x7f6cd4006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:14.756 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.464+0000 7f6ce4da6700 1 -- 172.21.15.52:0/522352797 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ce01039c0 msgr2=0x7f6ce0078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:14.756 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.464+0000 7f6ce4da6700 1 --2- 172.21.15.52:0/522352797 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ce01039c0 0x7f6ce0078d10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0000c00 tx=0x7f6cd0009250 comp rx=0 tx=0).stop 2023-12-16T20:45:14.756 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.464+0000 7f6ce4da6700 1 -- 172.21.15.52:0/522352797 shutdown_connections 2023-12-16T20:45:14.756 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.464+0000 7f6ce4da6700 1 --2- 172.21.15.52:0/522352797 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f6cc80362c0 0x7f6cc8038770 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:14.756 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.464+0000 7f6ce4da6700 1 --2- 172.21.15.52:0/522352797 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ce01039c0 0x7f6ce0078d10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:14.756 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.464+0000 7f6ce4da6700 1 -- 172.21.15.52:0/522352797 >> 172.21.15.52:0/522352797 conn(0x7f6ce00fda40 msgr2=0x7f6ce00ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:14.757 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.464+0000 7f6ce4da6700 1 -- 172.21.15.52:0/522352797 shutdown_connections 2023-12-16T20:45:14.757 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.464+0000 7f6ce4da6700 1 -- 172.21.15.52:0/522352797 wait complete. 2023-12-16T20:45:14.757 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying mgr service with default placement... 2023-12-16T20:45:14.848 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:14 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:14.848 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:14 smithi052 ceph-mon[80761]: Added host smithi052 2023-12-16T20:45:14.848 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:14 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:45:14.848 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:14 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:15.487 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-12-16T20:45:15.487 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.972+0000 7f9421442700 1 Processor -- start 2023-12-16T20:45:15.487 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.972+0000 7f9421442700 1 -- start start 2023-12-16T20:45:15.487 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.972+0000 7f9421442700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f941c1011d0 0x7f941c1015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:15.488 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.972+0000 7f9421442700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941c101b20 con 0x7f941c1011d0 2023-12-16T20:45:15.488 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.972+0000 7f941affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f941c1011d0 0x7f941c1015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:15.488 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.972+0000 7f941affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f941c1011d0 0x7f941c1015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45176/0 (socket says 172.21.15.52:45176) 2023-12-16T20:45:15.488 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.972+0000 7f941affd700 1 -- 172.21.15.52:0/1459811072 learned_addr learned my addr 172.21.15.52:0/1459811072 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:15.488 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.973+0000 7f941affd700 1 -- 172.21.15.52:0/1459811072 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941c101c60 con 0x7f941c1011d0 2023-12-16T20:45:15.488 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.973+0000 7f941affd700 1 --2- 172.21.15.52:0/1459811072 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f941c1011d0 0x7f941c1015e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9414009e50 tx=0x7f9414009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec45f2e3978664c1 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:15.488 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.973+0000 7f9419ffb700 1 -- 172.21.15.52:0/1459811072 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94140092e0 con 0x7f941c1011d0 2023-12-16T20:45:15.488 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.973+0000 7f9419ffb700 1 -- 172.21.15.52:0/1459811072 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f941400be90 con 0x7f941c1011d0 2023-12-16T20:45:15.488 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.973+0000 7f9419ffb700 1 -- 172.21.15.52:0/1459811072 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9414007400 con 0x7f941c1011d0 2023-12-16T20:45:15.488 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.973+0000 7f9421442700 1 -- 172.21.15.52:0/1459811072 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f941c1011d0 msgr2=0x7f941c1015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:15.488 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.973+0000 7f9421442700 1 --2- 172.21.15.52:0/1459811072 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f941c1011d0 0x7f941c1015e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9414009e50 tx=0x7f9414009840 comp rx=0 tx=0).stop 2023-12-16T20:45:15.488 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.974+0000 7f9421442700 1 -- 172.21.15.52:0/1459811072 shutdown_connections 2023-12-16T20:45:15.488 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.974+0000 7f9421442700 1 --2- 172.21.15.52:0/1459811072 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f941c1011d0 0x7f941c1015e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:15.488 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.974+0000 7f9421442700 1 -- 172.21.15.52:0/1459811072 >> 172.21.15.52:0/1459811072 conn(0x7f941c0fc970 msgr2=0x7f941c0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:15.489 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.974+0000 7f9421442700 1 -- 172.21.15.52:0/1459811072 shutdown_connections 2023-12-16T20:45:15.489 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.974+0000 7f9421442700 1 -- 172.21.15.52:0/1459811072 wait complete. 2023-12-16T20:45:15.489 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.974+0000 7f9421442700 1 Processor -- start 2023-12-16T20:45:15.489 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.974+0000 7f9421442700 1 -- start start 2023-12-16T20:45:15.489 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.975+0000 7f9421442700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f941c1011d0 0x7f941c1111c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:15.489 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.975+0000 7f9421442700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941c111700 con 0x7f941c1011d0 2023-12-16T20:45:15.489 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.975+0000 7f941affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f941c1011d0 0x7f941c1111c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:15.489 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.975+0000 7f941affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f941c1011d0 0x7f941c1111c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45188/0 (socket says 172.21.15.52:45188) 2023-12-16T20:45:15.489 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.975+0000 7f941affd700 1 -- 172.21.15.52:0/3916929308 learned_addr learned my addr 172.21.15.52:0/3916929308 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:15.489 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.975+0000 7f941affd700 1 -- 172.21.15.52:0/3916929308 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9414009ae0 con 0x7f941c1011d0 2023-12-16T20:45:15.489 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.975+0000 7f941affd700 1 --2- 172.21.15.52:0/3916929308 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f941c1011d0 0x7f941c1111c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f941400b740 tx=0x7f9414009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:15.489 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.975+0000 7f94137fe700 1 -- 172.21.15.52:0/3916929308 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9414007670 con 0x7f941c1011d0 2023-12-16T20:45:15.489 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.975+0000 7f94137fe700 1 -- 172.21.15.52:0/3916929308 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f9414007c40 con 0x7f941c1011d0 2023-12-16T20:45:15.489 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.975+0000 7f9421442700 1 -- 172.21.15.52:0/3916929308 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f941c111900 con 0x7f941c1011d0 2023-12-16T20:45:15.490 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.975+0000 7f94137fe700 1 -- 172.21.15.52:0/3916929308 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f941401cb10 con 0x7f941c1011d0 2023-12-16T20:45:15.490 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.976+0000 7f9421442700 1 -- 172.21.15.52:0/3916929308 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f941c1125a0 con 0x7f941c1011d0 2023-12-16T20:45:15.490 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.976+0000 7f94137fe700 1 -- 172.21.15.52:0/3916929308 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f9414019070 con 0x7f941c1011d0 2023-12-16T20:45:15.490 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.976+0000 7f94137fe700 1 --2- 172.21.15.52:0/3916929308 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f9408036640 0x7f9408038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:15.490 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.976+0000 7f94137fe700 1 -- 172.21.15.52:0/3916929308 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f9414049e50 con 0x7f941c1011d0 2023-12-16T20:45:15.490 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.976+0000 7f9421442700 1 -- 172.21.15.52:0/3916929308 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f941c05f270 con 0x7f941c1011d0 2023-12-16T20:45:15.490 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.977+0000 7f941a7fc700 1 --2- 172.21.15.52:0/3916929308 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f9408036640 0x7f9408038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:15.490 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.977+0000 7f941a7fc700 1 --2- 172.21.15.52:0/3916929308 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f9408036640 0x7f9408038af0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9404009990 tx=0x7f9404006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:15.490 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:14.981+0000 7f94137fe700 1 -- 172.21.15.52:0/3916929308 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9414017080 con 0x7f941c1011d0 2023-12-16T20:45:15.490 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.196+0000 7f9421442700 1 -- 172.21.15.52:0/3916929308 --> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f941c113050 con 0x7f9408036640 2023-12-16T20:45:15.490 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.201+0000 7f94137fe700 1 -- 172.21.15.52:0/3916929308 <== mgr.14120 v2:172.21.15.52:6800/221719009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f941c113050 con 0x7f9408036640 2023-12-16T20:45:15.490 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.203+0000 7f9421442700 1 -- 172.21.15.52:0/3916929308 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f9408036640 msgr2=0x7f9408038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:15.490 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.203+0000 7f9421442700 1 --2- 172.21.15.52:0/3916929308 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f9408036640 0x7f9408038af0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9404009990 tx=0x7f9404006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:15.490 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.203+0000 7f9421442700 1 -- 172.21.15.52:0/3916929308 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f941c1011d0 msgr2=0x7f941c1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:15.491 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.203+0000 7f9421442700 1 --2- 172.21.15.52:0/3916929308 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f941c1011d0 0x7f941c1111c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f941400b740 tx=0x7f9414009840 comp rx=0 tx=0).stop 2023-12-16T20:45:15.491 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.203+0000 7f9421442700 1 -- 172.21.15.52:0/3916929308 shutdown_connections 2023-12-16T20:45:15.491 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.203+0000 7f9421442700 1 --2- 172.21.15.52:0/3916929308 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f9408036640 0x7f9408038af0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:15.491 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.203+0000 7f9421442700 1 --2- 172.21.15.52:0/3916929308 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f941c1011d0 0x7f941c1111c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:15.491 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.203+0000 7f9421442700 1 -- 172.21.15.52:0/3916929308 >> 172.21.15.52:0/3916929308 conn(0x7f941c0fc970 msgr2=0x7f941c0fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:15.491 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.203+0000 7f9421442700 1 -- 172.21.15.52:0/3916929308 shutdown_connections 2023-12-16T20:45:15.491 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.203+0000 7f9421442700 1 -- 172.21.15.52:0/3916929308 wait complete. 2023-12-16T20:45:15.491 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying crash service with default placement... 2023-12-16T20:45:16.319 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2023-12-16T20:45:16.319 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.859+0000 7f50cee26700 1 Processor -- start 2023-12-16T20:45:16.319 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.859+0000 7f50cee26700 1 -- start start 2023-12-16T20:45:16.319 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.859+0000 7f50cee26700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50c8071000 0x7f50c8104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:16.319 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.859+0000 7f50cee26700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c81053e0 con 0x7f50c8071000 2023-12-16T20:45:16.319 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.860+0000 7f50ccbc2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50c8071000 0x7f50c8104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:16.319 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.860+0000 7f50ccbc2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50c8071000 0x7f50c8104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45204/0 (socket says 172.21.15.52:45204) 2023-12-16T20:45:16.319 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.860+0000 7f50ccbc2700 1 -- 172.21.15.52:0/1108708301 learned_addr learned my addr 172.21.15.52:0/1108708301 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.860+0000 7f50ccbc2700 1 -- 172.21.15.52:0/1108708301 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c8105520 con 0x7f50c8071000 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.860+0000 7f50ccbc2700 1 --2- 172.21.15.52:0/1108708301 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50c8071000 0x7f50c8104ea0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f50bc0098e0 tx=0x7f50bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa2aea64265ba8a5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.860+0000 7f50c77fe700 1 -- 172.21.15.52:0/1108708301 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50bc00b860 con 0x7f50c8071000 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.860+0000 7f50c77fe700 1 -- 172.21.15.52:0/1108708301 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f50bc00be30 con 0x7f50c8071000 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.860+0000 7f50c77fe700 1 -- 172.21.15.52:0/1108708301 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50bc010380 con 0x7f50c8071000 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.861+0000 7f50cee26700 1 -- 172.21.15.52:0/1108708301 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50c8071000 msgr2=0x7f50c8104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.861+0000 7f50cee26700 1 --2- 172.21.15.52:0/1108708301 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50c8071000 0x7f50c8104ea0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f50bc0098e0 tx=0x7f50bc009250 comp rx=0 tx=0).stop 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.861+0000 7f50cee26700 1 -- 172.21.15.52:0/1108708301 shutdown_connections 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.861+0000 7f50cee26700 1 --2- 172.21.15.52:0/1108708301 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50c8071000 0x7f50c8104ea0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.861+0000 7f50cee26700 1 -- 172.21.15.52:0/1108708301 >> 172.21.15.52:0/1108708301 conn(0x7f50c80fc970 msgr2=0x7f50c80fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.861+0000 7f50cee26700 1 -- 172.21.15.52:0/1108708301 shutdown_connections 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.861+0000 7f50cee26700 1 -- 172.21.15.52:0/1108708301 wait complete. 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.861+0000 7f50cee26700 1 Processor -- start 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.861+0000 7f50cee26700 1 -- start start 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.862+0000 7f50cee26700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50c8071000 0x7f50c8111160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:16.320 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.862+0000 7f50cee26700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c81116a0 con 0x7f50c8071000 2023-12-16T20:45:16.321 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.862+0000 7f50ccbc2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50c8071000 0x7f50c8111160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:16.321 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.862+0000 7f50ccbc2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50c8071000 0x7f50c8111160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45206/0 (socket says 172.21.15.52:45206) 2023-12-16T20:45:16.321 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.862+0000 7f50ccbc2700 1 -- 172.21.15.52:0/471128426 learned_addr learned my addr 172.21.15.52:0/471128426 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:16.321 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.862+0000 7f50ccbc2700 1 -- 172.21.15.52:0/471128426 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50bc0094f0 con 0x7f50c8071000 2023-12-16T20:45:16.321 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.862+0000 7f50ccbc2700 1 --2- 172.21.15.52:0/471128426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50c8071000 0x7f50c8111160 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f50bc006d40 tx=0x7f50bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:16.321 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.863+0000 7f50c57fa700 1 -- 172.21.15.52:0/471128426 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50bc0109b0 con 0x7f50c8071000 2023-12-16T20:45:16.321 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.863+0000 7f50cee26700 1 -- 172.21.15.52:0/471128426 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c81118a0 con 0x7f50c8071000 2023-12-16T20:45:16.321 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.863+0000 7f50c57fa700 1 -- 172.21.15.52:0/471128426 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f50bc024440 con 0x7f50c8071000 2023-12-16T20:45:16.321 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.863+0000 7f50c57fa700 1 -- 172.21.15.52:0/471128426 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50bc01a5f0 con 0x7f50c8071000 2023-12-16T20:45:16.322 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.863+0000 7f50cee26700 1 -- 172.21.15.52:0/471128426 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c8111cb0 con 0x7f50c8071000 2023-12-16T20:45:16.322 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.863+0000 7f50c57fa700 1 -- 172.21.15.52:0/471128426 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f50bc01a870 con 0x7f50c8071000 2023-12-16T20:45:16.322 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.863+0000 7f50cee26700 1 -- 172.21.15.52:0/471128426 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c804bb80 con 0x7f50c8071000 2023-12-16T20:45:16.322 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.864+0000 7f50c57fa700 1 --2- 172.21.15.52:0/471128426 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f50b80365f0 0x7f50b8038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:16.322 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.864+0000 7f50c57fa700 1 -- 172.21.15.52:0/471128426 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f50bc04a4f0 con 0x7f50c8071000 2023-12-16T20:45:16.322 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.864+0000 7f50c7fff700 1 --2- 172.21.15.52:0/471128426 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f50b80365f0 0x7f50b8038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:16.322 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.864+0000 7f50c7fff700 1 --2- 172.21.15.52:0/471128426 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f50b80365f0 0x7f50b8038aa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009990 tx=0x7f50b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:16.322 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.869+0000 7f50c57fa700 1 -- 172.21.15.52:0/471128426 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f50bc018b80 con 0x7f50c8071000 2023-12-16T20:45:16.322 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:15.989+0000 7f50c57fa700 1 -- 172.21.15.52:0/471128426 <== mon.0 v2:172.21.15.52:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50bc018b80 con 0x7f50c8071000 2023-12-16T20:45:16.322 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.082+0000 7f50cee26700 1 -- 172.21.15.52:0/471128426 --> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f50c80008d0 con 0x7f50b80365f0 2023-12-16T20:45:16.322 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.086+0000 7f50c57fa700 1 -- 172.21.15.52:0/471128426 <== mgr.14120 v2:172.21.15.52:6800/221719009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f50c80008d0 con 0x7f50b80365f0 2023-12-16T20:45:16.322 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.089+0000 7f50cee26700 1 -- 172.21.15.52:0/471128426 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f50b80365f0 msgr2=0x7f50b8038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:16.322 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.089+0000 7f50cee26700 1 --2- 172.21.15.52:0/471128426 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f50b80365f0 0x7f50b8038aa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009990 tx=0x7f50b4006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:16.322 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.089+0000 7f50cee26700 1 -- 172.21.15.52:0/471128426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50c8071000 msgr2=0x7f50c8111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:16.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.089+0000 7f50cee26700 1 --2- 172.21.15.52:0/471128426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50c8071000 0x7f50c8111160 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f50bc006d40 tx=0x7f50bc009250 comp rx=0 tx=0).stop 2023-12-16T20:45:16.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.089+0000 7f50cee26700 1 -- 172.21.15.52:0/471128426 shutdown_connections 2023-12-16T20:45:16.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.089+0000 7f50cee26700 1 --2- 172.21.15.52:0/471128426 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f50b80365f0 0x7f50b8038aa0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:16.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.089+0000 7f50cee26700 1 --2- 172.21.15.52:0/471128426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50c8071000 0x7f50c8111160 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:16.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.089+0000 7f50cee26700 1 -- 172.21.15.52:0/471128426 >> 172.21.15.52:0/471128426 conn(0x7f50c80fc970 msgr2=0x7f50c80fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:16.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.089+0000 7f50cee26700 1 -- 172.21.15.52:0/471128426 shutdown_connections 2023-12-16T20:45:16.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.089+0000 7f50cee26700 1 -- 172.21.15.52:0/471128426 wait complete. 2023-12-16T20:45:16.323 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying prometheus service with default placement... 2023-12-16T20:45:16.386 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:16 smithi052 ceph-mon[80761]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:16.386 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:16 smithi052 ceph-mon[80761]: Saving service mon spec with placement count:5 2023-12-16T20:45:16.386 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:16 smithi052 ceph-mon[80761]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:16.386 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:16 smithi052 ceph-mon[80761]: Saving service mgr spec with placement count:2 2023-12-16T20:45:16.386 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:16 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:16.386 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:16 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:16.386 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:16 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:17.010 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2023-12-16T20:45:17.011 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.561+0000 7f3ae57c4700 1 Processor -- start 2023-12-16T20:45:17.011 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.561+0000 7f3ae57c4700 1 -- start start 2023-12-16T20:45:17.011 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.561+0000 7f3ae57c4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ae01011d0 0x7f3ae01015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:17.011 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.561+0000 7f3ae57c4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae0101b20 con 0x7f3ae01011d0 2023-12-16T20:45:17.011 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.561+0000 7f3adeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ae01011d0 0x7f3ae01015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:17.011 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.561+0000 7f3adeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ae01011d0 0x7f3ae01015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45218/0 (socket says 172.21.15.52:45218) 2023-12-16T20:45:17.011 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.561+0000 7f3adeffd700 1 -- 172.21.15.52:0/4145588606 learned_addr learned my addr 172.21.15.52:0/4145588606 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:17.011 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.562+0000 7f3adeffd700 1 -- 172.21.15.52:0/4145588606 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae0101c60 con 0x7f3ae01011d0 2023-12-16T20:45:17.011 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.562+0000 7f3adeffd700 1 --2- 172.21.15.52:0/4145588606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ae01011d0 0x7f3ae01015e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40098e0 tx=0x7f3ad4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c147897efa0cd0cf server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:17.011 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.562+0000 7f3addffb700 1 -- 172.21.15.52:0/4145588606 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ad400b860 con 0x7f3ae01011d0 2023-12-16T20:45:17.011 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.562+0000 7f3addffb700 1 -- 172.21.15.52:0/4145588606 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ad400be70 con 0x7f3ae01011d0 2023-12-16T20:45:17.011 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.562+0000 7f3addffb700 1 -- 172.21.15.52:0/4145588606 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ad4010380 con 0x7f3ae01011d0 2023-12-16T20:45:17.011 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.563+0000 7f3ae57c4700 1 -- 172.21.15.52:0/4145588606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ae01011d0 msgr2=0x7f3ae01015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:17.012 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.563+0000 7f3ae57c4700 1 --2- 172.21.15.52:0/4145588606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ae01011d0 0x7f3ae01015e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40098e0 tx=0x7f3ad4009250 comp rx=0 tx=0).stop 2023-12-16T20:45:17.012 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.563+0000 7f3ae57c4700 1 -- 172.21.15.52:0/4145588606 shutdown_connections 2023-12-16T20:45:17.012 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.563+0000 7f3ae57c4700 1 --2- 172.21.15.52:0/4145588606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ae01011d0 0x7f3ae01015e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:17.012 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.563+0000 7f3ae57c4700 1 -- 172.21.15.52:0/4145588606 >> 172.21.15.52:0/4145588606 conn(0x7f3ae00fc950 msgr2=0x7f3ae00fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:17.012 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.563+0000 7f3ae57c4700 1 -- 172.21.15.52:0/4145588606 shutdown_connections 2023-12-16T20:45:17.012 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.563+0000 7f3ae57c4700 1 -- 172.21.15.52:0/4145588606 wait complete. 2023-12-16T20:45:17.012 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.563+0000 7f3ae57c4700 1 Processor -- start 2023-12-16T20:45:17.012 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.563+0000 7f3ae57c4700 1 -- start start 2023-12-16T20:45:17.012 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.564+0000 7f3ae57c4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ae01011d0 0x7f3ae0111180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:17.012 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.564+0000 7f3ae57c4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae01116c0 con 0x7f3ae01011d0 2023-12-16T20:45:17.012 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.564+0000 7f3adeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ae01011d0 0x7f3ae0111180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:17.012 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.564+0000 7f3adeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ae01011d0 0x7f3ae0111180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45232/0 (socket says 172.21.15.52:45232) 2023-12-16T20:45:17.012 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.564+0000 7f3adeffd700 1 -- 172.21.15.52:0/2127316260 learned_addr learned my addr 172.21.15.52:0/2127316260 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:17.012 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.564+0000 7f3adeffd700 1 -- 172.21.15.52:0/2127316260 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad40094f0 con 0x7f3ae01011d0 2023-12-16T20:45:17.013 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.564+0000 7f3adeffd700 1 --2- 172.21.15.52:0/2127316260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ae01011d0 0x7f3ae0111180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3ad400c010 tx=0x7f3ad4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:17.013 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.564+0000 7f3ac77fe700 1 -- 172.21.15.52:0/2127316260 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ad4010960 con 0x7f3ae01011d0 2023-12-16T20:45:17.013 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.564+0000 7f3ac77fe700 1 -- 172.21.15.52:0/2127316260 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ad401a430 con 0x7f3ae01011d0 2023-12-16T20:45:17.013 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.564+0000 7f3ae57c4700 1 -- 172.21.15.52:0/2127316260 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae01118c0 con 0x7f3ae01011d0 2023-12-16T20:45:17.013 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.564+0000 7f3ac77fe700 1 -- 172.21.15.52:0/2127316260 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ad4019580 con 0x7f3ae01011d0 2023-12-16T20:45:17.013 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.565+0000 7f3ae57c4700 1 -- 172.21.15.52:0/2127316260 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae0111cd0 con 0x7f3ae01011d0 2023-12-16T20:45:17.013 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.565+0000 7f3ac77fe700 1 -- 172.21.15.52:0/2127316260 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f3ad4019800 con 0x7f3ae01011d0 2023-12-16T20:45:17.013 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.565+0000 7f3ae57c4700 1 -- 172.21.15.52:0/2127316260 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae004bb80 con 0x7f3ae01011d0 2023-12-16T20:45:17.013 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.565+0000 7f3ac77fe700 1 --2- 172.21.15.52:0/2127316260 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f3ac80365f0 0x7f3ac8038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:17.013 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.566+0000 7f3ac77fe700 1 -- 172.21.15.52:0/2127316260 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3ad404a3c0 con 0x7f3ae01011d0 2023-12-16T20:45:17.013 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.566+0000 7f3ade7fc700 1 --2- 172.21.15.52:0/2127316260 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f3ac80365f0 0x7f3ac8038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:17.013 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.566+0000 7f3ade7fc700 1 --2- 172.21.15.52:0/2127316260 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f3ac80365f0 0x7f3ac8038aa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0009990 tx=0x7f3ad0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:17.013 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.570+0000 7f3ac77fe700 1 -- 172.21.15.52:0/2127316260 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3ad402d8c0 con 0x7f3ae01011d0 2023-12-16T20:45:17.014 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.786+0000 7f3ae57c4700 1 -- 172.21.15.52:0/2127316260 --> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ae00febc0 con 0x7f3ac80365f0 2023-12-16T20:45:17.014 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.791+0000 7f3ac77fe700 1 -- 172.21.15.52:0/2127316260 <== mgr.14120 v2:172.21.15.52:6800/221719009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f3ae00febc0 con 0x7f3ac80365f0 2023-12-16T20:45:17.014 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.793+0000 7f3ae57c4700 1 -- 172.21.15.52:0/2127316260 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f3ac80365f0 msgr2=0x7f3ac8038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:17.014 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.793+0000 7f3ae57c4700 1 --2- 172.21.15.52:0/2127316260 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f3ac80365f0 0x7f3ac8038aa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0009990 tx=0x7f3ad0006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:17.014 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.793+0000 7f3ae57c4700 1 -- 172.21.15.52:0/2127316260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ae01011d0 msgr2=0x7f3ae0111180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:17.014 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.793+0000 7f3ae57c4700 1 --2- 172.21.15.52:0/2127316260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ae01011d0 0x7f3ae0111180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3ad400c010 tx=0x7f3ad4009250 comp rx=0 tx=0).stop 2023-12-16T20:45:17.014 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.793+0000 7f3ae57c4700 1 -- 172.21.15.52:0/2127316260 shutdown_connections 2023-12-16T20:45:17.014 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.793+0000 7f3ae57c4700 1 --2- 172.21.15.52:0/2127316260 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f3ac80365f0 0x7f3ac8038aa0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:17.014 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.793+0000 7f3ae57c4700 1 --2- 172.21.15.52:0/2127316260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ae01011d0 0x7f3ae0111180 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:17.014 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.793+0000 7f3ae57c4700 1 -- 172.21.15.52:0/2127316260 >> 172.21.15.52:0/2127316260 conn(0x7f3ae00fc950 msgr2=0x7f3ae00767f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:17.014 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.793+0000 7f3ae57c4700 1 -- 172.21.15.52:0/2127316260 shutdown_connections 2023-12-16T20:45:17.014 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:16.793+0000 7f3ae57c4700 1 -- 172.21.15.52:0/2127316260 wait complete. 2023-12-16T20:45:17.014 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying grafana service with default placement... 2023-12-16T20:45:17.665 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:17 smithi052 ceph-mon[80761]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:17.665 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:17 smithi052 ceph-mon[80761]: Saving service crash spec with placement * 2023-12-16T20:45:17.665 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:17 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:17.665 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:17 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:17.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2023-12-16T20:45:17.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.444+0000 7f210a7a8700 1 Processor -- start 2023-12-16T20:45:17.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.444+0000 7f210a7a8700 1 -- start start 2023-12-16T20:45:17.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.445+0000 7f210a7a8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2104103970 0x7f2104103d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:17.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.445+0000 7f210a7a8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21041042c0 con 0x7f2104103970 2023-12-16T20:45:17.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.445+0000 7f2103fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2104103970 0x7f2104103d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:17.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.445+0000 7f2103fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2104103970 0x7f2104103d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45244/0 (socket says 172.21.15.52:45244) 2023-12-16T20:45:17.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.445+0000 7f2103fff700 1 -- 172.21.15.52:0/4126472918 learned_addr learned my addr 172.21.15.52:0/4126472918 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:17.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.445+0000 7f2103fff700 1 -- 172.21.15.52:0/4126472918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2104104400 con 0x7f2104103970 2023-12-16T20:45:17.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.445+0000 7f2103fff700 1 --2- 172.21.15.52:0/4126472918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2104103970 0x7f2104103d80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0098e0 tx=0x7f20ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d77329b261f8 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:17.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.445+0000 7f2102ffd700 1 -- 172.21.15.52:0/4126472918 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20ec00b860 con 0x7f2104103970 2023-12-16T20:45:17.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.445+0000 7f2102ffd700 1 -- 172.21.15.52:0/4126472918 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20ec00be70 con 0x7f2104103970 2023-12-16T20:45:17.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.445+0000 7f2102ffd700 1 -- 172.21.15.52:0/4126472918 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20ec010380 con 0x7f2104103970 2023-12-16T20:45:17.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.446+0000 7f210a7a8700 1 -- 172.21.15.52:0/4126472918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2104103970 msgr2=0x7f2104103d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:17.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.446+0000 7f210a7a8700 1 --2- 172.21.15.52:0/4126472918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2104103970 0x7f2104103d80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0098e0 tx=0x7f20ec009250 comp rx=0 tx=0).stop 2023-12-16T20:45:17.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.446+0000 7f210a7a8700 1 -- 172.21.15.52:0/4126472918 shutdown_connections 2023-12-16T20:45:17.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.446+0000 7f210a7a8700 1 --2- 172.21.15.52:0/4126472918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2104103970 0x7f2104103d80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:17.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.447+0000 7f210a7a8700 1 -- 172.21.15.52:0/4126472918 >> 172.21.15.52:0/4126472918 conn(0x7f21040fc0b0 msgr2=0x7f21040fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:17.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.447+0000 7f210a7a8700 1 -- 172.21.15.52:0/4126472918 shutdown_connections 2023-12-16T20:45:17.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.447+0000 7f210a7a8700 1 -- 172.21.15.52:0/4126472918 wait complete. 2023-12-16T20:45:17.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.447+0000 7f210a7a8700 1 Processor -- start 2023-12-16T20:45:17.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.447+0000 7f210a7a8700 1 -- start start 2023-12-16T20:45:17.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.447+0000 7f210a7a8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2104103970 0x7f21041031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:17.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.447+0000 7f210a7a8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2104101630 con 0x7f2104103970 2023-12-16T20:45:17.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.448+0000 7f2103fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2104103970 0x7f21041031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:17.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.448+0000 7f2103fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2104103970 0x7f21041031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45256/0 (socket says 172.21.15.52:45256) 2023-12-16T20:45:17.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.448+0000 7f2103fff700 1 -- 172.21.15.52:0/3676594316 learned_addr learned my addr 172.21.15.52:0/3676594316 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:17.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.448+0000 7f2103fff700 1 -- 172.21.15.52:0/3676594316 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ec0094f0 con 0x7f2104103970 2023-12-16T20:45:17.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.448+0000 7f2103fff700 1 --2- 172.21.15.52:0/3676594316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2104103970 0x7f21041031b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00b440 tx=0x7f20ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:17.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.448+0000 7f2100ff9700 1 -- 172.21.15.52:0/3676594316 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20ec010850 con 0x7f2104103970 2023-12-16T20:45:17.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.448+0000 7f2100ff9700 1 -- 172.21.15.52:0/3676594316 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20ec010e60 con 0x7f2104103970 2023-12-16T20:45:17.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.448+0000 7f210a7a8700 1 -- 172.21.15.52:0/3676594316 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2104101830 con 0x7f2104103970 2023-12-16T20:45:17.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.448+0000 7f210a7a8700 1 -- 172.21.15.52:0/3676594316 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2104101c40 con 0x7f2104103970 2023-12-16T20:45:17.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.448+0000 7f2100ff9700 1 -- 172.21.15.52:0/3676594316 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20ec01aea0 con 0x7f2104103970 2023-12-16T20:45:17.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.449+0000 7f2100ff9700 1 -- 172.21.15.52:0/3676594316 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f20ec02d3f0 con 0x7f2104103970 2023-12-16T20:45:17.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.449+0000 7f210a7a8700 1 -- 172.21.15.52:0/3676594316 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f210404bb80 con 0x7f2104103970 2023-12-16T20:45:17.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.449+0000 7f2100ff9700 1 --2- 172.21.15.52:0/3676594316 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f20f0036640 0x7f20f0038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:17.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.450+0000 7f2100ff9700 1 -- 172.21.15.52:0/3676594316 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f20ec04a1a0 con 0x7f2104103970 2023-12-16T20:45:17.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.450+0000 7f21037fe700 1 --2- 172.21.15.52:0/3676594316 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f20f0036640 0x7f20f0038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:17.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.450+0000 7f21037fe700 1 --2- 172.21.15.52:0/3676594316 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f20f0036640 0x7f20f0038af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009990 tx=0x7f20f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:17.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.454+0000 7f2100ff9700 1 -- 172.21.15.52:0/3676594316 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f20ec02b8e0 con 0x7f2104103970 2023-12-16T20:45:17.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.663+0000 7f210a7a8700 1 -- 172.21.15.52:0/3676594316 --> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f210410ae50 con 0x7f20f0036640 2023-12-16T20:45:17.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.668+0000 7f2100ff9700 1 -- 172.21.15.52:0/3676594316 <== mgr.14120 v2:172.21.15.52:6800/221719009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f210410ae50 con 0x7f20f0036640 2023-12-16T20:45:17.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.670+0000 7f210a7a8700 1 -- 172.21.15.52:0/3676594316 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f20f0036640 msgr2=0x7f20f0038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:17.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.670+0000 7f210a7a8700 1 --2- 172.21.15.52:0/3676594316 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f20f0036640 0x7f20f0038af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009990 tx=0x7f20f4006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:17.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.670+0000 7f210a7a8700 1 -- 172.21.15.52:0/3676594316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2104103970 msgr2=0x7f21041031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:17.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.670+0000 7f210a7a8700 1 --2- 172.21.15.52:0/3676594316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2104103970 0x7f21041031b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00b440 tx=0x7f20ec009250 comp rx=0 tx=0).stop 2023-12-16T20:45:17.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.670+0000 7f210a7a8700 1 -- 172.21.15.52:0/3676594316 shutdown_connections 2023-12-16T20:45:17.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.670+0000 7f210a7a8700 1 --2- 172.21.15.52:0/3676594316 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f20f0036640 0x7f20f0038af0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:17.923 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.670+0000 7f210a7a8700 1 --2- 172.21.15.52:0/3676594316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2104103970 0x7f21041031b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:17.923 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.670+0000 7f210a7a8700 1 -- 172.21.15.52:0/3676594316 >> 172.21.15.52:0/3676594316 conn(0x7f21040fc0b0 msgr2=0x7f21040fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:17.923 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.670+0000 7f210a7a8700 1 -- 172.21.15.52:0/3676594316 shutdown_connections 2023-12-16T20:45:17.923 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:17.671+0000 7f210a7a8700 1 -- 172.21.15.52:0/3676594316 wait complete. 2023-12-16T20:45:17.923 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying node-exporter service with default placement... 2023-12-16T20:45:18.455 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:18 smithi052 ceph-mon[80761]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:18.455 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:18 smithi052 ceph-mon[80761]: Saving service prometheus spec with placement count:1 2023-12-16T20:45:18.455 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:18 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:18.455 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:18 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:18.455 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:18 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:18.652 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2023-12-16T20:45:18.652 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.179+0000 7fa6f4b3a700 1 Processor -- start 2023-12-16T20:45:18.652 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.179+0000 7fa6f4b3a700 1 -- start start 2023-12-16T20:45:18.652 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.180+0000 7fa6f4b3a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6f0071000 0x7fa6f0104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:18.652 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.180+0000 7fa6f4b3a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f01053e0 con 0x7fa6f0071000 2023-12-16T20:45:18.652 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.180+0000 7fa6ee59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6f0071000 0x7fa6f0104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:18.652 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.180+0000 7fa6ee59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6f0071000 0x7fa6f0104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45258/0 (socket says 172.21.15.52:45258) 2023-12-16T20:45:18.653 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.180+0000 7fa6ee59c700 1 -- 172.21.15.52:0/4040077632 learned_addr learned my addr 172.21.15.52:0/4040077632 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:18.653 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.180+0000 7fa6ee59c700 1 -- 172.21.15.52:0/4040077632 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f0105520 con 0x7fa6f0071000 2023-12-16T20:45:18.653 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.181+0000 7fa6ee59c700 1 --2- 172.21.15.52:0/4040077632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6f0071000 0x7fa6f0104ea0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8009ed0 tx=0x7fa6e8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3018f83f2e40b3d8 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:18.653 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.181+0000 7fa6ed59a700 1 -- 172.21.15.52:0/4040077632 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e800fe90 con 0x7fa6f0071000 2023-12-16T20:45:18.653 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.181+0000 7fa6ed59a700 1 -- 172.21.15.52:0/4040077632 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6e80178b0 con 0x7fa6f0071000 2023-12-16T20:45:18.653 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.181+0000 7fa6ed59a700 1 -- 172.21.15.52:0/4040077632 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e8007a00 con 0x7fa6f0071000 2023-12-16T20:45:18.653 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.181+0000 7fa6f4b3a700 1 -- 172.21.15.52:0/4040077632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6f0071000 msgr2=0x7fa6f0104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:18.653 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.181+0000 7fa6f4b3a700 1 --2- 172.21.15.52:0/4040077632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6f0071000 0x7fa6f0104ea0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8009ed0 tx=0x7fa6e8009840 comp rx=0 tx=0).stop 2023-12-16T20:45:18.653 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.182+0000 7fa6f4b3a700 1 -- 172.21.15.52:0/4040077632 shutdown_connections 2023-12-16T20:45:18.653 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.182+0000 7fa6f4b3a700 1 --2- 172.21.15.52:0/4040077632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6f0071000 0x7fa6f0104ea0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:18.653 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.182+0000 7fa6f4b3a700 1 -- 172.21.15.52:0/4040077632 >> 172.21.15.52:0/4040077632 conn(0x7fa6f00fc970 msgr2=0x7fa6f00fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:18.653 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.182+0000 7fa6f4b3a700 1 -- 172.21.15.52:0/4040077632 shutdown_connections 2023-12-16T20:45:18.653 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.182+0000 7fa6f4b3a700 1 -- 172.21.15.52:0/4040077632 wait complete. 2023-12-16T20:45:18.653 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.182+0000 7fa6f4b3a700 1 Processor -- start 2023-12-16T20:45:18.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.182+0000 7fa6f4b3a700 1 -- start start 2023-12-16T20:45:18.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.183+0000 7fa6f4b3a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6f0071000 0x7fa6f010efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:18.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.183+0000 7fa6f4b3a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f010f4e0 con 0x7fa6f0071000 2023-12-16T20:45:18.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.183+0000 7fa6ee59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6f0071000 0x7fa6f010efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:18.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.183+0000 7fa6ee59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6f0071000 0x7fa6f010efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45268/0 (socket says 172.21.15.52:45268) 2023-12-16T20:45:18.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.183+0000 7fa6ee59c700 1 -- 172.21.15.52:0/1044164640 learned_addr learned my addr 172.21.15.52:0/1044164640 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:18.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.183+0000 7fa6ee59c700 1 -- 172.21.15.52:0/1044164640 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e8009ae0 con 0x7fa6f0071000 2023-12-16T20:45:18.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.183+0000 7fa6ee59c700 1 --2- 172.21.15.52:0/1044164640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6f0071000 0x7fa6f010efa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8007440 tx=0x7fa6e8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:18.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.183+0000 7fa6e6ffd700 1 -- 172.21.15.52:0/1044164640 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e8020670 con 0x7fa6f0071000 2023-12-16T20:45:18.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.183+0000 7fa6e6ffd700 1 -- 172.21.15.52:0/1044164640 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6e800fe90 con 0x7fa6f0071000 2023-12-16T20:45:18.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.183+0000 7fa6e6ffd700 1 -- 172.21.15.52:0/1044164640 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e801fa60 con 0x7fa6f0071000 2023-12-16T20:45:18.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.184+0000 7fa6f4b3a700 1 -- 172.21.15.52:0/1044164640 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f010f6e0 con 0x7fa6f0071000 2023-12-16T20:45:18.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.184+0000 7fa6f4b3a700 1 -- 172.21.15.52:0/1044164640 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f0110380 con 0x7fa6f0071000 2023-12-16T20:45:18.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.184+0000 7fa6e6ffd700 1 -- 172.21.15.52:0/1044164640 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fa6e801fbc0 con 0x7fa6f0071000 2023-12-16T20:45:18.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.185+0000 7fa6e6ffd700 1 --2- 172.21.15.52:0/1044164640 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fa6dc036640 0x7fa6dc038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:18.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.185+0000 7fa6e6ffd700 1 -- 172.21.15.52:0/1044164640 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa6e804edd0 con 0x7fa6f0071000 2023-12-16T20:45:18.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.185+0000 7fa6edd9b700 1 --2- 172.21.15.52:0/1044164640 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fa6dc036640 0x7fa6dc038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:18.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.185+0000 7fa6f4b3a700 1 -- 172.21.15.52:0/1044164640 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f010fb50 con 0x7fa6f0071000 2023-12-16T20:45:18.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.185+0000 7fa6edd9b700 1 --2- 172.21.15.52:0/1044164640 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fa6dc036640 0x7fa6dc038af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009990 tx=0x7fa6d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:18.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.194+0000 7fa6e6ffd700 1 -- 172.21.15.52:0/1044164640 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa6e802e020 con 0x7fa6f0071000 2023-12-16T20:45:18.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.405+0000 7fa6f4b3a700 1 -- 172.21.15.52:0/1044164640 --> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6f0110680 con 0x7fa6dc036640 2023-12-16T20:45:18.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.409+0000 7fa6e6ffd700 1 -- 172.21.15.52:0/1044164640 <== mgr.14120 v2:172.21.15.52:6800/221719009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fa6f0110680 con 0x7fa6dc036640 2023-12-16T20:45:18.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.411+0000 7fa6f4b3a700 1 -- 172.21.15.52:0/1044164640 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fa6dc036640 msgr2=0x7fa6dc038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:18.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.411+0000 7fa6f4b3a700 1 --2- 172.21.15.52:0/1044164640 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fa6dc036640 0x7fa6dc038af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009990 tx=0x7fa6d8006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:18.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.411+0000 7fa6f4b3a700 1 -- 172.21.15.52:0/1044164640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6f0071000 msgr2=0x7fa6f010efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:18.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.411+0000 7fa6f4b3a700 1 --2- 172.21.15.52:0/1044164640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6f0071000 0x7fa6f010efa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8007440 tx=0x7fa6e8009840 comp rx=0 tx=0).stop 2023-12-16T20:45:18.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.411+0000 7fa6f4b3a700 1 -- 172.21.15.52:0/1044164640 shutdown_connections 2023-12-16T20:45:18.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.411+0000 7fa6f4b3a700 1 --2- 172.21.15.52:0/1044164640 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7fa6dc036640 0x7fa6dc038af0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:18.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.411+0000 7fa6f4b3a700 1 --2- 172.21.15.52:0/1044164640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6f0071000 0x7fa6f010efa0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:18.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.411+0000 7fa6f4b3a700 1 -- 172.21.15.52:0/1044164640 >> 172.21.15.52:0/1044164640 conn(0x7fa6f00fc970 msgr2=0x7fa6f0103180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:18.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.411+0000 7fa6f4b3a700 1 -- 172.21.15.52:0/1044164640 shutdown_connections 2023-12-16T20:45:18.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.411+0000 7fa6f4b3a700 1 -- 172.21.15.52:0/1044164640 wait complete. 2023-12-16T20:45:18.656 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying alertmanager service with default placement... 2023-12-16T20:45:19.552 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2023-12-16T20:45:19.552 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.953+0000 7f95abda5700 1 Processor -- start 2023-12-16T20:45:19.552 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.953+0000 7f95abda5700 1 -- start start 2023-12-16T20:45:19.552 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.953+0000 7f95abda5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95a4076c60 0x7f95a4104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:19.552 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.953+0000 7f95abda5700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a41053e0 con 0x7f95a4076c60 2023-12-16T20:45:19.552 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.953+0000 7f95a9b41700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95a4076c60 0x7f95a4104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:19.552 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.953+0000 7f95a9b41700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95a4076c60 0x7f95a4104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45284/0 (socket says 172.21.15.52:45284) 2023-12-16T20:45:19.552 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.953+0000 7f95a9b41700 1 -- 172.21.15.52:0/3464312448 learned_addr learned my addr 172.21.15.52:0/3464312448 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:19.553 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.954+0000 7f95a9b41700 1 -- 172.21.15.52:0/3464312448 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a4105520 con 0x7f95a4076c60 2023-12-16T20:45:19.553 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.954+0000 7f95a9b41700 1 --2- 172.21.15.52:0/3464312448 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95a4076c60 0x7f95a4104ea0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f95a00098e0 tx=0x7f95a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b672d60b80bc8a29 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:19.553 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.954+0000 7f95a8b3f700 1 -- 172.21.15.52:0/3464312448 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95a000b860 con 0x7f95a4076c60 2023-12-16T20:45:19.553 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.954+0000 7f95a8b3f700 1 -- 172.21.15.52:0/3464312448 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95a000be70 con 0x7f95a4076c60 2023-12-16T20:45:19.553 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.954+0000 7f95a8b3f700 1 -- 172.21.15.52:0/3464312448 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95a0010380 con 0x7f95a4076c60 2023-12-16T20:45:19.553 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.955+0000 7f95abda5700 1 -- 172.21.15.52:0/3464312448 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95a4076c60 msgr2=0x7f95a4104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:19.553 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.955+0000 7f95abda5700 1 --2- 172.21.15.52:0/3464312448 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95a4076c60 0x7f95a4104ea0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f95a00098e0 tx=0x7f95a0009250 comp rx=0 tx=0).stop 2023-12-16T20:45:19.553 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.955+0000 7f95abda5700 1 -- 172.21.15.52:0/3464312448 shutdown_connections 2023-12-16T20:45:19.553 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.955+0000 7f95abda5700 1 --2- 172.21.15.52:0/3464312448 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95a4076c60 0x7f95a4104ea0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:19.553 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.955+0000 7f95abda5700 1 -- 172.21.15.52:0/3464312448 >> 172.21.15.52:0/3464312448 conn(0x7f95a40fc950 msgr2=0x7f95a40fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:19.553 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.955+0000 7f95abda5700 1 -- 172.21.15.52:0/3464312448 shutdown_connections 2023-12-16T20:45:19.553 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.955+0000 7f95abda5700 1 -- 172.21.15.52:0/3464312448 wait complete. 2023-12-16T20:45:19.554 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.955+0000 7f95abda5700 1 Processor -- start 2023-12-16T20:45:19.554 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.955+0000 7f95abda5700 1 -- start start 2023-12-16T20:45:19.554 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.956+0000 7f95abda5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95a4076c60 0x7f95a4102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:19.554 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.956+0000 7f95abda5700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a4100da0 con 0x7f95a4076c60 2023-12-16T20:45:19.554 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.956+0000 7f95a9b41700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95a4076c60 0x7f95a4102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:19.554 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.956+0000 7f95a9b41700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95a4076c60 0x7f95a4102920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45292/0 (socket says 172.21.15.52:45292) 2023-12-16T20:45:19.554 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.956+0000 7f95a9b41700 1 -- 172.21.15.52:0/785161120 learned_addr learned my addr 172.21.15.52:0/785161120 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:19.554 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.956+0000 7f95a9b41700 1 -- 172.21.15.52:0/785161120 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a00094f0 con 0x7f95a4076c60 2023-12-16T20:45:19.554 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.956+0000 7f95a9b41700 1 --2- 172.21.15.52:0/785161120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95a4076c60 0x7f95a4102920 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f95a0006d40 tx=0x7f95a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:19.554 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.956+0000 7f959a7fc700 1 -- 172.21.15.52:0/785161120 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95a0010960 con 0x7f95a4076c60 2023-12-16T20:45:19.554 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.956+0000 7f959a7fc700 1 -- 172.21.15.52:0/785161120 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95a001a430 con 0x7f95a4076c60 2023-12-16T20:45:19.554 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.956+0000 7f95abda5700 1 -- 172.21.15.52:0/785161120 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a4100fa0 con 0x7f95a4076c60 2023-12-16T20:45:19.554 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.956+0000 7f959a7fc700 1 -- 172.21.15.52:0/785161120 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95a00245b0 con 0x7f95a4076c60 2023-12-16T20:45:19.554 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.956+0000 7f95abda5700 1 -- 172.21.15.52:0/785161120 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a41013b0 con 0x7f95a4076c60 2023-12-16T20:45:19.554 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.957+0000 7f959a7fc700 1 -- 172.21.15.52:0/785161120 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f95a002d3f0 con 0x7f95a4076c60 2023-12-16T20:45:19.555 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.957+0000 7f959a7fc700 1 --2- 172.21.15.52:0/785161120 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f9590036640 0x7f9590038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:19.555 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.957+0000 7f95abda5700 1 -- 172.21.15.52:0/785161120 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a410b180 con 0x7f95a4076c60 2023-12-16T20:45:19.555 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.957+0000 7f959a7fc700 1 -- 172.21.15.52:0/785161120 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f95a004a0d0 con 0x7f95a4076c60 2023-12-16T20:45:19.555 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.957+0000 7f95a9340700 1 --2- 172.21.15.52:0/785161120 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f9590036640 0x7f9590038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:19.555 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.958+0000 7f95a9340700 1 --2- 172.21.15.52:0/785161120 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f9590036640 0x7f9590038af0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9594009990 tx=0x7f9594006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:19.555 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:18.962+0000 7f959a7fc700 1 -- 172.21.15.52:0/785161120 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f95a002b8e0 con 0x7f95a4076c60 2023-12-16T20:45:19.555 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.173+0000 7f95abda5700 1 -- 172.21.15.52:0/785161120 --> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f95a402b150 con 0x7f9590036640 2023-12-16T20:45:19.555 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.178+0000 7f959a7fc700 1 -- 172.21.15.52:0/785161120 <== mgr.14120 v2:172.21.15.52:6800/221719009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f95a402b150 con 0x7f9590036640 2023-12-16T20:45:19.555 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.180+0000 7f95abda5700 1 -- 172.21.15.52:0/785161120 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f9590036640 msgr2=0x7f9590038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:19.555 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.180+0000 7f95abda5700 1 --2- 172.21.15.52:0/785161120 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f9590036640 0x7f9590038af0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9594009990 tx=0x7f9594006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:19.555 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.180+0000 7f95abda5700 1 -- 172.21.15.52:0/785161120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95a4076c60 msgr2=0x7f95a4102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:19.555 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.180+0000 7f95abda5700 1 --2- 172.21.15.52:0/785161120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95a4076c60 0x7f95a4102920 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f95a0006d40 tx=0x7f95a0009250 comp rx=0 tx=0).stop 2023-12-16T20:45:19.555 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.180+0000 7f95abda5700 1 -- 172.21.15.52:0/785161120 shutdown_connections 2023-12-16T20:45:19.555 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.180+0000 7f95abda5700 1 --2- 172.21.15.52:0/785161120 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f9590036640 0x7f9590038af0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:19.555 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.180+0000 7f95abda5700 1 --2- 172.21.15.52:0/785161120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95a4076c60 0x7f95a4102920 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:19.556 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.180+0000 7f95abda5700 1 -- 172.21.15.52:0/785161120 >> 172.21.15.52:0/785161120 conn(0x7f95a40fc950 msgr2=0x7f95a40fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:19.556 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.180+0000 7f95abda5700 1 -- 172.21.15.52:0/785161120 shutdown_connections 2023-12-16T20:45:19.556 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.180+0000 7f95abda5700 1 -- 172.21.15.52:0/785161120 wait complete. 2023-12-16T20:45:19.619 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:19 smithi052 ceph-mon[80761]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:19.619 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:19 smithi052 ceph-mon[80761]: Saving service grafana spec with placement count:1 2023-12-16T20:45:19.619 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:19 smithi052 ceph-mon[80761]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:19.619 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:19 smithi052 ceph-mon[80761]: Saving service node-exporter spec with placement * 2023-12-16T20:45:19.619 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:19 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:19.619 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:19 smithi052 ceph-mon[80761]: from='mgr.14120 172.21.15.52:0/327920741' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:20.245 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.795+0000 7f5d7a554700 1 Processor -- start 2023-12-16T20:45:20.245 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.795+0000 7f5d7a554700 1 -- start start 2023-12-16T20:45:20.245 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.795+0000 7f5d7a554700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d74076be0 0x7f5d74107320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:20.246 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.795+0000 7f5d7a554700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d74102d10 con 0x7f5d74076be0 2023-12-16T20:45:20.246 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.795+0000 7f5d73fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d74076be0 0x7f5d74107320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:20.246 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.795+0000 7f5d73fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d74076be0 0x7f5d74107320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45298/0 (socket says 172.21.15.52:45298) 2023-12-16T20:45:20.246 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.795+0000 7f5d73fff700 1 -- 172.21.15.52:0/3379298251 learned_addr learned my addr 172.21.15.52:0/3379298251 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:20.246 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.795+0000 7f5d73fff700 1 -- 172.21.15.52:0/3379298251 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d74107860 con 0x7f5d74076be0 2023-12-16T20:45:20.246 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.796+0000 7f5d73fff700 1 --2- 172.21.15.52:0/3379298251 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d74076be0 0x7f5d74107320 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5d640098e0 tx=0x7f5d64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeaea5ed5d673f86 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:20.246 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.796+0000 7f5d72ffd700 1 -- 172.21.15.52:0/3379298251 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d6400b860 con 0x7f5d74076be0 2023-12-16T20:45:20.246 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.796+0000 7f5d72ffd700 1 -- 172.21.15.52:0/3379298251 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d6400be70 con 0x7f5d74076be0 2023-12-16T20:45:20.246 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.796+0000 7f5d72ffd700 1 -- 172.21.15.52:0/3379298251 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d64010380 con 0x7f5d74076be0 2023-12-16T20:45:20.246 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.796+0000 7f5d7a554700 1 -- 172.21.15.52:0/3379298251 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d74076be0 msgr2=0x7f5d74107320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:20.246 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.796+0000 7f5d7a554700 1 --2- 172.21.15.52:0/3379298251 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d74076be0 0x7f5d74107320 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5d640098e0 tx=0x7f5d64009250 comp rx=0 tx=0).stop 2023-12-16T20:45:20.246 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.796+0000 7f5d7a554700 1 -- 172.21.15.52:0/3379298251 shutdown_connections 2023-12-16T20:45:20.246 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.796+0000 7f5d7a554700 1 --2- 172.21.15.52:0/3379298251 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d74076be0 0x7f5d74107320 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:20.246 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.796+0000 7f5d7a554700 1 -- 172.21.15.52:0/3379298251 >> 172.21.15.52:0/3379298251 conn(0x7f5d740fc950 msgr2=0x7f5d740fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:20.247 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.797+0000 7f5d7a554700 1 -- 172.21.15.52:0/3379298251 shutdown_connections 2023-12-16T20:45:20.247 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.797+0000 7f5d7a554700 1 -- 172.21.15.52:0/3379298251 wait complete. 2023-12-16T20:45:20.247 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.797+0000 7f5d7a554700 1 Processor -- start 2023-12-16T20:45:20.247 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.797+0000 7f5d7a554700 1 -- start start 2023-12-16T20:45:20.247 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.797+0000 7f5d7a554700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d74076be0 0x7f5d741133c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:20.247 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.797+0000 7f5d7a554700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d74113900 con 0x7f5d74076be0 2023-12-16T20:45:20.247 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.797+0000 7f5d73fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d74076be0 0x7f5d741133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:20.247 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.797+0000 7f5d73fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d74076be0 0x7f5d741133c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45308/0 (socket says 172.21.15.52:45308) 2023-12-16T20:45:20.247 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.797+0000 7f5d73fff700 1 -- 172.21.15.52:0/2494286708 learned_addr learned my addr 172.21.15.52:0/2494286708 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:20.247 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.798+0000 7f5d73fff700 1 -- 172.21.15.52:0/2494286708 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d640094f0 con 0x7f5d74076be0 2023-12-16T20:45:20.247 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.798+0000 7f5d73fff700 1 --2- 172.21.15.52:0/2494286708 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d74076be0 0x7f5d741133c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5d64006d40 tx=0x7f5d64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:20.247 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.798+0000 7f5d70ff9700 1 -- 172.21.15.52:0/2494286708 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d64010960 con 0x7f5d74076be0 2023-12-16T20:45:20.247 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.798+0000 7f5d70ff9700 1 -- 172.21.15.52:0/2494286708 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d6401a430 con 0x7f5d74076be0 2023-12-16T20:45:20.247 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.798+0000 7f5d7a554700 1 -- 172.21.15.52:0/2494286708 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d74113b00 con 0x7f5d74076be0 2023-12-16T20:45:20.247 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.798+0000 7f5d70ff9700 1 -- 172.21.15.52:0/2494286708 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d640245b0 con 0x7f5d74076be0 2023-12-16T20:45:20.248 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.798+0000 7f5d7a554700 1 -- 172.21.15.52:0/2494286708 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d74113f10 con 0x7f5d74076be0 2023-12-16T20:45:20.248 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.799+0000 7f5d70ff9700 1 -- 172.21.15.52:0/2494286708 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f5d64010ac0 con 0x7f5d74076be0 2023-12-16T20:45:20.248 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.799+0000 7f5d70ff9700 1 --2- 172.21.15.52:0/2494286708 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f5d5c0365f0 0x7f5d5c038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:20.248 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.799+0000 7f5d70ff9700 1 -- 172.21.15.52:0/2494286708 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5d6404a100 con 0x7f5d74076be0 2023-12-16T20:45:20.248 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.799+0000 7f5d7a554700 1 -- 172.21.15.52:0/2494286708 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7404bb80 con 0x7f5d74076be0 2023-12-16T20:45:20.248 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.799+0000 7f5d737fe700 1 --2- 172.21.15.52:0/2494286708 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f5d5c0365f0 0x7f5d5c038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:20.248 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.800+0000 7f5d737fe700 1 --2- 172.21.15.52:0/2494286708 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f5d5c0365f0 0x7f5d5c038aa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009990 tx=0x7f5d68006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:20.248 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.804+0000 7f5d70ff9700 1 -- 172.21.15.52:0/2494286708 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5d6401a9f0 con 0x7f5d74076be0 2023-12-16T20:45:20.248 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:19.999+0000 7f5d7a554700 1 -- 172.21.15.52:0/2494286708 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f5d7410ceb0 con 0x7f5d74076be0 2023-12-16T20:45:20.248 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.003+0000 7f5d70ff9700 1 -- 172.21.15.52:0/2494286708 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f5d6402b2f0 con 0x7f5d74076be0 2023-12-16T20:45:20.248 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.005+0000 7f5d7a554700 1 -- 172.21.15.52:0/2494286708 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f5d5c0365f0 msgr2=0x7f5d5c038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:20.248 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.005+0000 7f5d7a554700 1 --2- 172.21.15.52:0/2494286708 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f5d5c0365f0 0x7f5d5c038aa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009990 tx=0x7f5d68006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:20.248 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.005+0000 7f5d7a554700 1 -- 172.21.15.52:0/2494286708 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d74076be0 msgr2=0x7f5d741133c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:20.248 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.005+0000 7f5d7a554700 1 --2- 172.21.15.52:0/2494286708 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d74076be0 0x7f5d741133c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5d64006d40 tx=0x7f5d64009250 comp rx=0 tx=0).stop 2023-12-16T20:45:20.248 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.005+0000 7f5d7a554700 1 -- 172.21.15.52:0/2494286708 shutdown_connections 2023-12-16T20:45:20.249 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.005+0000 7f5d7a554700 1 --2- 172.21.15.52:0/2494286708 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f5d5c0365f0 0x7f5d5c038aa0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:20.249 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.005+0000 7f5d7a554700 1 --2- 172.21.15.52:0/2494286708 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d74076be0 0x7f5d741133c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:20.249 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.005+0000 7f5d7a554700 1 -- 172.21.15.52:0/2494286708 >> 172.21.15.52:0/2494286708 conn(0x7f5d740fc950 msgr2=0x7f5d740fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:20.249 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.005+0000 7f5d7a554700 1 -- 172.21.15.52:0/2494286708 shutdown_connections 2023-12-16T20:45:20.249 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.005+0000 7f5d7a554700 1 -- 172.21.15.52:0/2494286708 wait complete. 2023-12-16T20:45:20.691 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:20 smithi052 ceph-mon[80761]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:20.692 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:20 smithi052 ceph-mon[80761]: Saving service alertmanager spec with placement count:1 2023-12-16T20:45:20.692 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:20 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/2494286708' entity='client.admin' 2023-12-16T20:45:20.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.487+0000 7f43a709f700 1 Processor -- start 2023-12-16T20:45:20.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.487+0000 7f43a709f700 1 -- start start 2023-12-16T20:45:20.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.487+0000 7f43a709f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43a0078dc0 0x7f43a00771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:20.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.487+0000 7f43a709f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a0077710 con 0x7f43a0078dc0 2023-12-16T20:45:20.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.487+0000 7f43a4e3b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43a0078dc0 0x7f43a00771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:20.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.487+0000 7f43a4e3b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43a0078dc0 0x7f43a00771d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45318/0 (socket says 172.21.15.52:45318) 2023-12-16T20:45:20.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.487+0000 7f43a4e3b700 1 -- 172.21.15.52:0/2523717908 learned_addr learned my addr 172.21.15.52:0/2523717908 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:20.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.487+0000 7f43a4e3b700 1 -- 172.21.15.52:0/2523717908 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a0077850 con 0x7f43a0078dc0 2023-12-16T20:45:20.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.487+0000 7f43a4e3b700 1 --2- 172.21.15.52:0/2523717908 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43a0078dc0 0x7f43a00771d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f43940098e0 tx=0x7f4394009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e41cdb3eb48f2b8 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:20.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.488+0000 7f439f7fe700 1 -- 172.21.15.52:0/2523717908 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f439400b860 con 0x7f43a0078dc0 2023-12-16T20:45:20.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.488+0000 7f439f7fe700 1 -- 172.21.15.52:0/2523717908 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f439400be70 con 0x7f43a0078dc0 2023-12-16T20:45:20.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.488+0000 7f439f7fe700 1 -- 172.21.15.52:0/2523717908 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4394010380 con 0x7f43a0078dc0 2023-12-16T20:45:20.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.488+0000 7f43a709f700 1 -- 172.21.15.52:0/2523717908 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43a0078dc0 msgr2=0x7f43a00771d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:20.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.488+0000 7f43a709f700 1 --2- 172.21.15.52:0/2523717908 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43a0078dc0 0x7f43a00771d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f43940098e0 tx=0x7f4394009250 comp rx=0 tx=0).stop 2023-12-16T20:45:20.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.488+0000 7f43a709f700 1 -- 172.21.15.52:0/2523717908 shutdown_connections 2023-12-16T20:45:20.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.488+0000 7f43a709f700 1 --2- 172.21.15.52:0/2523717908 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43a0078dc0 0x7f43a00771d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:20.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.488+0000 7f43a709f700 1 -- 172.21.15.52:0/2523717908 >> 172.21.15.52:0/2523717908 conn(0x7f43a00fda40 msgr2=0x7f43a00ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:20.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.488+0000 7f43a709f700 1 -- 172.21.15.52:0/2523717908 shutdown_connections 2023-12-16T20:45:20.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.488+0000 7f43a709f700 1 -- 172.21.15.52:0/2523717908 wait complete. 2023-12-16T20:45:20.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.489+0000 7f43a709f700 1 Processor -- start 2023-12-16T20:45:20.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.489+0000 7f43a709f700 1 -- start start 2023-12-16T20:45:20.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.489+0000 7f43a709f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43a0078dc0 0x7f43a0113380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:20.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.489+0000 7f43a709f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a01138c0 con 0x7f43a0078dc0 2023-12-16T20:45:20.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.489+0000 7f43a4e3b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43a0078dc0 0x7f43a0113380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:20.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.489+0000 7f43a4e3b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43a0078dc0 0x7f43a0113380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45330/0 (socket says 172.21.15.52:45330) 2023-12-16T20:45:20.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.489+0000 7f43a4e3b700 1 -- 172.21.15.52:0/3729962760 learned_addr learned my addr 172.21.15.52:0/3729962760 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:20.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.490+0000 7f43a4e3b700 1 -- 172.21.15.52:0/3729962760 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43940094f0 con 0x7f43a0078dc0 2023-12-16T20:45:20.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.490+0000 7f43a4e3b700 1 --2- 172.21.15.52:0/3729962760 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43a0078dc0 0x7f43a0113380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f43940094c0 tx=0x7f4394009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:20.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.490+0000 7f439d7fa700 1 -- 172.21.15.52:0/3729962760 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4394010850 con 0x7f43a0078dc0 2023-12-16T20:45:20.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.490+0000 7f439d7fa700 1 -- 172.21.15.52:0/3729962760 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4394010e60 con 0x7f43a0078dc0 2023-12-16T20:45:20.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.490+0000 7f43a709f700 1 -- 172.21.15.52:0/3729962760 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a0113ac0 con 0x7f43a0078dc0 2023-12-16T20:45:20.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.490+0000 7f439d7fa700 1 -- 172.21.15.52:0/3729962760 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43940243f0 con 0x7f43a0078dc0 2023-12-16T20:45:20.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.490+0000 7f43a709f700 1 -- 172.21.15.52:0/3729962760 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a0113ed0 con 0x7f43a0078dc0 2023-12-16T20:45:20.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.491+0000 7f439d7fa700 1 -- 172.21.15.52:0/3729962760 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f43940246b0 con 0x7f43a0078dc0 2023-12-16T20:45:20.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.491+0000 7f43a709f700 1 -- 172.21.15.52:0/3729962760 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a0108620 con 0x7f43a0078dc0 2023-12-16T20:45:20.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.491+0000 7f439d7fa700 1 --2- 172.21.15.52:0/3729962760 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f43900365f0 0x7f4390038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:20.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.491+0000 7f439d7fa700 1 -- 172.21.15.52:0/3729962760 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f439404a730 con 0x7f43a0078dc0 2023-12-16T20:45:20.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.491+0000 7f439ffff700 1 --2- 172.21.15.52:0/3729962760 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f43900365f0 0x7f4390038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:20.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.491+0000 7f439ffff700 1 --2- 172.21.15.52:0/3729962760 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f43900365f0 0x7f4390038aa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f438c009990 tx=0x7f438c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:20.923 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.496+0000 7f439d7fa700 1 -- 172.21.15.52:0/3729962760 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4394018b80 con 0x7f43a0078dc0 2023-12-16T20:45:20.923 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.690+0000 7f43a709f700 1 -- 172.21.15.52:0/3729962760 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f43a004bb80 con 0x7f43a0078dc0 2023-12-16T20:45:20.923 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.694+0000 7f439d7fa700 1 -- 172.21.15.52:0/3729962760 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f4394018350 con 0x7f43a0078dc0 2023-12-16T20:45:20.923 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.696+0000 7f43a709f700 1 -- 172.21.15.52:0/3729962760 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f43900365f0 msgr2=0x7f4390038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:20.923 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.696+0000 7f43a709f700 1 --2- 172.21.15.52:0/3729962760 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f43900365f0 0x7f4390038aa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f438c009990 tx=0x7f438c006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:20.923 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.696+0000 7f43a709f700 1 -- 172.21.15.52:0/3729962760 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43a0078dc0 msgr2=0x7f43a0113380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:20.923 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.696+0000 7f43a709f700 1 --2- 172.21.15.52:0/3729962760 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43a0078dc0 0x7f43a0113380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f43940094c0 tx=0x7f4394009250 comp rx=0 tx=0).stop 2023-12-16T20:45:20.923 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.697+0000 7f43a709f700 1 -- 172.21.15.52:0/3729962760 shutdown_connections 2023-12-16T20:45:20.924 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.697+0000 7f43a709f700 1 --2- 172.21.15.52:0/3729962760 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f43900365f0 0x7f4390038aa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:20.924 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.697+0000 7f43a709f700 1 --2- 172.21.15.52:0/3729962760 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43a0078dc0 0x7f43a0113380 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:20.924 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.697+0000 7f43a709f700 1 -- 172.21.15.52:0/3729962760 >> 172.21.15.52:0/3729962760 conn(0x7f43a00fda40 msgr2=0x7f43a00ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:20.924 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.697+0000 7f43a709f700 1 -- 172.21.15.52:0/3729962760 shutdown_connections 2023-12-16T20:45:20.924 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:20.697+0000 7f43a709f700 1 -- 172.21.15.52:0/3729962760 wait complete. 2023-12-16T20:45:20.924 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling the dashboard module... 2023-12-16T20:45:22.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:21 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3729962760' entity='client.admin' 2023-12-16T20:45:22.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:21 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/2025537842' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-12-16T20:45:22.038 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.167+0000 7ff8e8ab8700 1 Processor -- start 2023-12-16T20:45:22.038 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.167+0000 7ff8e8ab8700 1 -- start start 2023-12-16T20:45:22.038 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.167+0000 7ff8e8ab8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff8e41039c0 0x7ff8e4103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:22.038 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.167+0000 7ff8e8ab8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e4104310 con 0x7ff8e41039c0 2023-12-16T20:45:22.038 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.167+0000 7ff8e259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff8e41039c0 0x7ff8e4103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:22.038 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.167+0000 7ff8e259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff8e41039c0 0x7ff8e4103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45342/0 (socket says 172.21.15.52:45342) 2023-12-16T20:45:22.038 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.167+0000 7ff8e259c700 1 -- 172.21.15.52:0/2165556208 learned_addr learned my addr 172.21.15.52:0/2165556208 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:22.038 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.168+0000 7ff8e259c700 1 -- 172.21.15.52:0/2165556208 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e4101fb0 con 0x7ff8e41039c0 2023-12-16T20:45:22.039 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.168+0000 7ff8e259c700 1 --2- 172.21.15.52:0/2165556208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff8e41039c0 0x7ff8e4103dd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40098e0 tx=0x7ff8d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c958f41eb9f226 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:22.039 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.168+0000 7ff8e159a700 1 -- 172.21.15.52:0/2165556208 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8d400b860 con 0x7ff8e41039c0 2023-12-16T20:45:22.039 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.168+0000 7ff8e159a700 1 -- 172.21.15.52:0/2165556208 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8d400be70 con 0x7ff8e41039c0 2023-12-16T20:45:22.039 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.168+0000 7ff8e159a700 1 -- 172.21.15.52:0/2165556208 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8d4010380 con 0x7ff8e41039c0 2023-12-16T20:45:22.039 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.169+0000 7ff8e8ab8700 1 -- 172.21.15.52:0/2165556208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff8e41039c0 msgr2=0x7ff8e4103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:22.039 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.169+0000 7ff8e8ab8700 1 --2- 172.21.15.52:0/2165556208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff8e41039c0 0x7ff8e4103dd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40098e0 tx=0x7ff8d4009250 comp rx=0 tx=0).stop 2023-12-16T20:45:22.039 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.169+0000 7ff8e8ab8700 1 -- 172.21.15.52:0/2165556208 shutdown_connections 2023-12-16T20:45:22.039 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.169+0000 7ff8e8ab8700 1 --2- 172.21.15.52:0/2165556208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff8e41039c0 0x7ff8e4103dd0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:22.039 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.169+0000 7ff8e8ab8700 1 -- 172.21.15.52:0/2165556208 >> 172.21.15.52:0/2165556208 conn(0x7ff8e40fda40 msgr2=0x7ff8e40ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:22.039 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.169+0000 7ff8e8ab8700 1 -- 172.21.15.52:0/2165556208 shutdown_connections 2023-12-16T20:45:22.039 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.169+0000 7ff8e8ab8700 1 -- 172.21.15.52:0/2165556208 wait complete. 2023-12-16T20:45:22.039 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.169+0000 7ff8e8ab8700 1 Processor -- start 2023-12-16T20:45:22.039 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.169+0000 7ff8e8ab8700 1 -- start start 2023-12-16T20:45:22.039 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.170+0000 7ff8e8ab8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff8e41039c0 0x7ff8e4111130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:22.039 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.170+0000 7ff8e8ab8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e4111670 con 0x7ff8e41039c0 2023-12-16T20:45:22.040 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.170+0000 7ff8e259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff8e41039c0 0x7ff8e4111130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:22.040 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.170+0000 7ff8e259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff8e41039c0 0x7ff8e4111130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45348/0 (socket says 172.21.15.52:45348) 2023-12-16T20:45:22.040 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.170+0000 7ff8e259c700 1 -- 172.21.15.52:0/2025537842 learned_addr learned my addr 172.21.15.52:0/2025537842 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:22.040 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.170+0000 7ff8e259c700 1 -- 172.21.15.52:0/2025537842 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d40094f0 con 0x7ff8e41039c0 2023-12-16T20:45:22.040 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.170+0000 7ff8e259c700 1 --2- 172.21.15.52:0/2025537842 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff8e41039c0 0x7ff8e4111130 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4003080 tx=0x7ff8d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:22.040 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.170+0000 7ff8d2ffd700 1 -- 172.21.15.52:0/2025537842 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8d4010800 con 0x7ff8e41039c0 2023-12-16T20:45:22.040 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.170+0000 7ff8d2ffd700 1 -- 172.21.15.52:0/2025537842 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8d4010e10 con 0x7ff8e41039c0 2023-12-16T20:45:22.040 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.170+0000 7ff8e8ab8700 1 -- 172.21.15.52:0/2025537842 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e4111870 con 0x7ff8e41039c0 2023-12-16T20:45:22.040 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.170+0000 7ff8d2ffd700 1 -- 172.21.15.52:0/2025537842 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8d401a430 con 0x7ff8e41039c0 2023-12-16T20:45:22.040 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.170+0000 7ff8e8ab8700 1 -- 172.21.15.52:0/2025537842 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e4111c80 con 0x7ff8e41039c0 2023-12-16T20:45:22.040 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.171+0000 7ff8d2ffd700 1 -- 172.21.15.52:0/2025537842 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7ff8d4010960 con 0x7ff8e41039c0 2023-12-16T20:45:22.040 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.171+0000 7ff8d2ffd700 1 --2- 172.21.15.52:0/2025537842 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7ff8cc0365f0 0x7ff8cc038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:22.040 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.171+0000 7ff8d2ffd700 1 -- 172.21.15.52:0/2025537842 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff8d404a5b0 con 0x7ff8e41039c0 2023-12-16T20:45:22.040 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.171+0000 7ff8e8ab8700 1 -- 172.21.15.52:0/2025537842 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e4102d30 con 0x7ff8e41039c0 2023-12-16T20:45:22.040 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.171+0000 7ff8e1d9b700 1 --2- 172.21.15.52:0/2025537842 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7ff8cc0365f0 0x7ff8cc038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:22.041 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.172+0000 7ff8e1d9b700 1 --2- 172.21.15.52:0/2025537842 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7ff8cc0365f0 0x7ff8cc038aa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009990 tx=0x7ff8d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:22.041 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.176+0000 7ff8d2ffd700 1 -- 172.21.15.52:0/2025537842 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff8d402d3f0 con 0x7ff8e41039c0 2023-12-16T20:45:22.041 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.429+0000 7ff8e8ab8700 1 -- 172.21.15.52:0/2025537842 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7ff8e4112950 con 0x7ff8e41039c0 2023-12-16T20:45:22.041 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.693+0000 7ff8d2ffd700 1 -- 172.21.15.52:0/2025537842 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7ff8d400fda0 con 0x7ff8e41039c0 2023-12-16T20:45:22.041 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.695+0000 7ff8d2ffd700 1 -- 172.21.15.52:0/2025537842 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7ff8d4024440 con 0x7ff8e41039c0 2023-12-16T20:45:22.041 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.697+0000 7ff8e8ab8700 1 -- 172.21.15.52:0/2025537842 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7ff8cc0365f0 msgr2=0x7ff8cc038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:22.041 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.697+0000 7ff8e8ab8700 1 --2- 172.21.15.52:0/2025537842 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7ff8cc0365f0 0x7ff8cc038aa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009990 tx=0x7ff8d8006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:22.041 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.697+0000 7ff8e8ab8700 1 -- 172.21.15.52:0/2025537842 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff8e41039c0 msgr2=0x7ff8e4111130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:22.041 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.697+0000 7ff8e8ab8700 1 --2- 172.21.15.52:0/2025537842 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff8e41039c0 0x7ff8e4111130 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4003080 tx=0x7ff8d4009250 comp rx=0 tx=0).stop 2023-12-16T20:45:22.041 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.698+0000 7ff8e8ab8700 1 -- 172.21.15.52:0/2025537842 shutdown_connections 2023-12-16T20:45:22.041 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.698+0000 7ff8e8ab8700 1 --2- 172.21.15.52:0/2025537842 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7ff8cc0365f0 0x7ff8cc038aa0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:22.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.698+0000 7ff8e8ab8700 1 --2- 172.21.15.52:0/2025537842 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff8e41039c0 0x7ff8e4111130 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:22.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.698+0000 7ff8e8ab8700 1 -- 172.21.15.52:0/2025537842 >> 172.21.15.52:0/2025537842 conn(0x7ff8e40fda40 msgr2=0x7ff8e40ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:22.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.698+0000 7ff8e8ab8700 1 -- 172.21.15.52:0/2025537842 shutdown_connections 2023-12-16T20:45:22.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:21.698+0000 7ff8e8ab8700 1 -- 172.21.15.52:0/2025537842 wait complete. 2023-12-16T20:45:22.835 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:22 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/2025537842' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-12-16T20:45:22.835 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:22 smithi052 ceph-mon[80761]: mgrmap e9: smithi052.nmkcbs(active, since 14s) 2023-12-16T20:45:22.835 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:22 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/4101499987' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-16T20:45:22.838 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2023-12-16T20:45:22.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-12-16T20:45:22.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2023-12-16T20:45:22.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "active_name": "smithi052.nmkcbs", 2023-12-16T20:45:22.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-12-16T20:45:22.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2023-12-16T20:45:22.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.291+0000 7f62a6a12700 1 Processor -- start 2023-12-16T20:45:22.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.291+0000 7f62a6a12700 1 -- start start 2023-12-16T20:45:22.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.292+0000 7f62a6a12700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62a0070fa0 0x7f62a0104f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:22.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.292+0000 7f62a6a12700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a0105440 con 0x7f62a0070fa0 2023-12-16T20:45:22.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.292+0000 7f629ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62a0070fa0 0x7f62a0104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:22.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.292+0000 7f629ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62a0070fa0 0x7f62a0104f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45366/0 (socket says 172.21.15.52:45366) 2023-12-16T20:45:22.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.292+0000 7f629ffff700 1 -- 172.21.15.52:0/2233266663 learned_addr learned my addr 172.21.15.52:0/2233266663 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:22.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.292+0000 7f629ffff700 1 -- 172.21.15.52:0/2233266663 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a0105580 con 0x7f62a0070fa0 2023-12-16T20:45:22.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.293+0000 7f629ffff700 1 --2- 172.21.15.52:0/2233266663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62a0070fa0 0x7f62a0104f00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f62900098e0 tx=0x7f6290009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc8ae5b913b722b6 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:22.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.293+0000 7f629effd700 1 -- 172.21.15.52:0/2233266663 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f629000b860 con 0x7f62a0070fa0 2023-12-16T20:45:22.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.293+0000 7f629effd700 1 -- 172.21.15.52:0/2233266663 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f629000be70 con 0x7f62a0070fa0 2023-12-16T20:45:22.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.293+0000 7f629effd700 1 -- 172.21.15.52:0/2233266663 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6290010380 con 0x7f62a0070fa0 2023-12-16T20:45:22.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.293+0000 7f62a6a12700 1 -- 172.21.15.52:0/2233266663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62a0070fa0 msgr2=0x7f62a0104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:22.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.293+0000 7f62a6a12700 1 --2- 172.21.15.52:0/2233266663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62a0070fa0 0x7f62a0104f00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f62900098e0 tx=0x7f6290009250 comp rx=0 tx=0).stop 2023-12-16T20:45:22.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.293+0000 7f62a6a12700 1 -- 172.21.15.52:0/2233266663 shutdown_connections 2023-12-16T20:45:22.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.293+0000 7f62a6a12700 1 --2- 172.21.15.52:0/2233266663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62a0070fa0 0x7f62a0104f00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:22.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.293+0000 7f62a6a12700 1 -- 172.21.15.52:0/2233266663 >> 172.21.15.52:0/2233266663 conn(0x7f62a00fc970 msgr2=0x7f62a00fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:22.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.294+0000 7f62a6a12700 1 -- 172.21.15.52:0/2233266663 shutdown_connections 2023-12-16T20:45:22.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.294+0000 7f62a6a12700 1 -- 172.21.15.52:0/2233266663 wait complete. 2023-12-16T20:45:22.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.294+0000 7f62a6a12700 1 Processor -- start 2023-12-16T20:45:22.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.294+0000 7f62a6a12700 1 -- start start 2023-12-16T20:45:22.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.294+0000 7f62a6a12700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62a0070fa0 0x7f62a01111e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:22.846 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.294+0000 7f62a6a12700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a0111720 con 0x7f62a0070fa0 2023-12-16T20:45:22.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.295+0000 7f629ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62a0070fa0 0x7f62a01111e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:22.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.295+0000 7f629ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62a0070fa0 0x7f62a01111e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45382/0 (socket says 172.21.15.52:45382) 2023-12-16T20:45:22.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.295+0000 7f629ffff700 1 -- 172.21.15.52:0/4101499987 learned_addr learned my addr 172.21.15.52:0/4101499987 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:22.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.295+0000 7f629ffff700 1 -- 172.21.15.52:0/4101499987 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62900094f0 con 0x7f62a0070fa0 2023-12-16T20:45:22.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.295+0000 7f629ffff700 1 --2- 172.21.15.52:0/4101499987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62a0070fa0 0x7f62a01111e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6290006d40 tx=0x7f6290009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:22.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.295+0000 7f629cff9700 1 -- 172.21.15.52:0/4101499987 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6290010960 con 0x7f62a0070fa0 2023-12-16T20:45:22.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.295+0000 7f629cff9700 1 -- 172.21.15.52:0/4101499987 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f629001a430 con 0x7f62a0070fa0 2023-12-16T20:45:22.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.295+0000 7f62a6a12700 1 -- 172.21.15.52:0/4101499987 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a0111920 con 0x7f62a0070fa0 2023-12-16T20:45:22.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.295+0000 7f62a6a12700 1 -- 172.21.15.52:0/4101499987 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a0111d30 con 0x7f62a0070fa0 2023-12-16T20:45:22.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.296+0000 7f629cff9700 1 -- 172.21.15.52:0/4101499987 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62900245b0 con 0x7f62a0070fa0 2023-12-16T20:45:22.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.296+0000 7f629cff9700 1 -- 172.21.15.52:0/4101499987 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f629002d3f0 con 0x7f62a0070fa0 2023-12-16T20:45:22.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.296+0000 7f62a6a12700 1 -- 172.21.15.52:0/4101499987 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a004bb80 con 0x7f62a0070fa0 2023-12-16T20:45:22.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.297+0000 7f629cff9700 1 --2- 172.21.15.52:0/4101499987 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f6288036690 0x7f6288038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:22.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.297+0000 7f629f7fe700 1 -- 172.21.15.52:0/4101499987 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f6288036690 msgr2=0x7f6288038b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/221719009 2023-12-16T20:45:22.847 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.297+0000 7f629cff9700 1 -- 172.21.15.52:0/4101499987 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f629004a130 con 0x7f62a0070fa0 2023-12-16T20:45:22.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.297+0000 7f629f7fe700 1 --2- 172.21.15.52:0/4101499987 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f6288036690 0x7f6288038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T20:45:22.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.301+0000 7f629cff9700 1 -- 172.21.15.52:0/4101499987 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f629002b6b0 con 0x7f62a0070fa0 2023-12-16T20:45:22.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.498+0000 7f629f7fe700 1 -- 172.21.15.52:0/4101499987 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f6288036690 msgr2=0x7f6288038b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/221719009 2023-12-16T20:45:22.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.498+0000 7f629f7fe700 1 --2- 172.21.15.52:0/4101499987 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f6288036690 0x7f6288038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T20:45:22.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.555+0000 7f62a6a12700 1 -- 172.21.15.52:0/4101499987 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f62a010af80 con 0x7f62a0070fa0 2023-12-16T20:45:22.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.556+0000 7f629cff9700 1 -- 172.21.15.52:0/4101499987 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f62900179c0 con 0x7f62a0070fa0 2023-12-16T20:45:22.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.558+0000 7f62a6a12700 1 -- 172.21.15.52:0/4101499987 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f6288036690 msgr2=0x7f6288038b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T20:45:22.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.558+0000 7f62a6a12700 1 --2- 172.21.15.52:0/4101499987 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f6288036690 0x7f6288038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:22.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.558+0000 7f62a6a12700 1 -- 172.21.15.52:0/4101499987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62a0070fa0 msgr2=0x7f62a01111e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:22.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.558+0000 7f62a6a12700 1 --2- 172.21.15.52:0/4101499987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62a0070fa0 0x7f62a01111e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6290006d40 tx=0x7f6290009250 comp rx=0 tx=0).stop 2023-12-16T20:45:22.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.558+0000 7f62a6a12700 1 -- 172.21.15.52:0/4101499987 shutdown_connections 2023-12-16T20:45:22.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.558+0000 7f62a6a12700 1 --2- 172.21.15.52:0/4101499987 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f6288036690 0x7f6288038b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:22.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.558+0000 7f62a6a12700 1 --2- 172.21.15.52:0/4101499987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62a0070fa0 0x7f62a01111e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:22.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.558+0000 7f62a6a12700 1 -- 172.21.15.52:0/4101499987 >> 172.21.15.52:0/4101499987 conn(0x7f62a00fc970 msgr2=0x7f62a00fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:22.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.559+0000 7f62a6a12700 1 -- 172.21.15.52:0/4101499987 shutdown_connections 2023-12-16T20:45:22.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:22.559+0000 7f62a6a12700 1 -- 172.21.15.52:0/4101499987 wait complete. 2023-12-16T20:45:22.849 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for the mgr to restart... 2023-12-16T20:45:22.849 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr epoch 9... 2023-12-16T20:45:29.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:28 smithi052 ceph-mon[80761]: Active manager daemon smithi052.nmkcbs restarted 2023-12-16T20:45:29.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:28 smithi052 ceph-mon[80761]: Activating manager daemon smithi052.nmkcbs 2023-12-16T20:45:29.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:28 smithi052 ceph-mon[80761]: osdmap e3: 0 total, 0 up, 0 in 2023-12-16T20:45:29.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:28 smithi052 ceph-mon[80761]: mgrmap e10: smithi052.nmkcbs(active, starting, since 0.00311514s) 2023-12-16T20:45:29.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:28 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2023-12-16T20:45:29.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:28 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr metadata", "who": "smithi052.nmkcbs", "id": "smithi052.nmkcbs"}]: dispatch 2023-12-16T20:45:29.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:28 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T20:45:29.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:28 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T20:45:29.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:28 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T20:45:29.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:28 smithi052 ceph-mon[80761]: Manager daemon smithi052.nmkcbs is now available 2023-12-16T20:45:29.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:28 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:45:29.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:28 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:45:29.836 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:29 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.nmkcbs/mirror_snapshot_schedule"}]: dispatch 2023-12-16T20:45:29.836 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:29 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.nmkcbs/trash_purge_schedule"}]: dispatch 2023-12-16T20:45:29.836 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:29 smithi052 ceph-mon[80761]: mgrmap e11: smithi052.nmkcbs(active, since 1.00564s) 2023-12-16T20:45:29.837 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:29 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:29.839 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2023-12-16T20:45:29.839 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-12-16T20:45:29.839 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "initialized": true 2023-12-16T20:45:29.839 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2023-12-16T20:45:29.839 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.110+0000 7f7e1275e700 1 Processor -- start 2023-12-16T20:45:29.839 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.110+0000 7f7e1275e700 1 -- start start 2023-12-16T20:45:29.839 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.110+0000 7f7e1275e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e0c070fc0 0x7f7e0c104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:29.839 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.110+0000 7f7e1275e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c105420 con 0x7f7e0c070fc0 2023-12-16T20:45:29.840 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.111+0000 7f7e0bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e0c070fc0 0x7f7e0c104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:29.840 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.111+0000 7f7e0bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e0c070fc0 0x7f7e0c104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58036/0 (socket says 172.21.15.52:58036) 2023-12-16T20:45:29.843 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.111+0000 7f7e0bfff700 1 -- 172.21.15.52:0/2169637271 learned_addr learned my addr 172.21.15.52:0/2169637271 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:29.843 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.111+0000 7f7e0bfff700 1 -- 172.21.15.52:0/2169637271 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0c105560 con 0x7f7e0c070fc0 2023-12-16T20:45:29.843 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.111+0000 7f7e0bfff700 1 --2- 172.21.15.52:0/2169637271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e0c070fc0 0x7f7e0c104ee0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc0098e0 tx=0x7f7dfc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=858cbd169f738089 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:29.843 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.111+0000 7f7e0affd700 1 -- 172.21.15.52:0/2169637271 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dfc00b860 con 0x7f7e0c070fc0 2023-12-16T20:45:29.843 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.111+0000 7f7e0affd700 1 -- 172.21.15.52:0/2169637271 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7dfc00be70 con 0x7f7e0c070fc0 2023-12-16T20:45:29.843 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.111+0000 7f7e0affd700 1 -- 172.21.15.52:0/2169637271 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dfc010380 con 0x7f7e0c070fc0 2023-12-16T20:45:29.843 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.112+0000 7f7e1275e700 1 -- 172.21.15.52:0/2169637271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e0c070fc0 msgr2=0x7f7e0c104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:29.843 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.112+0000 7f7e1275e700 1 --2- 172.21.15.52:0/2169637271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e0c070fc0 0x7f7e0c104ee0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc0098e0 tx=0x7f7dfc009250 comp rx=0 tx=0).stop 2023-12-16T20:45:29.844 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.112+0000 7f7e1275e700 1 -- 172.21.15.52:0/2169637271 shutdown_connections 2023-12-16T20:45:29.844 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.112+0000 7f7e1275e700 1 --2- 172.21.15.52:0/2169637271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e0c070fc0 0x7f7e0c104ee0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:29.844 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.112+0000 7f7e1275e700 1 -- 172.21.15.52:0/2169637271 >> 172.21.15.52:0/2169637271 conn(0x7f7e0c0fc970 msgr2=0x7f7e0c0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:29.844 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.112+0000 7f7e1275e700 1 -- 172.21.15.52:0/2169637271 shutdown_connections 2023-12-16T20:45:29.844 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.112+0000 7f7e1275e700 1 -- 172.21.15.52:0/2169637271 wait complete. 2023-12-16T20:45:29.844 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.113+0000 7f7e1275e700 1 Processor -- start 2023-12-16T20:45:29.844 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.113+0000 7f7e1275e700 1 -- start start 2023-12-16T20:45:29.844 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.113+0000 7f7e1275e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e0c070fc0 0x7f7e0c1111c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:29.844 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.113+0000 7f7e1275e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c111700 con 0x7f7e0c070fc0 2023-12-16T20:45:29.844 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.113+0000 7f7e0bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e0c070fc0 0x7f7e0c1111c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:29.844 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.113+0000 7f7e0bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e0c070fc0 0x7f7e0c1111c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58052/0 (socket says 172.21.15.52:58052) 2023-12-16T20:45:29.844 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.113+0000 7f7e0bfff700 1 -- 172.21.15.52:0/1956508725 learned_addr learned my addr 172.21.15.52:0/1956508725 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:29.844 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.114+0000 7f7e0bfff700 1 -- 172.21.15.52:0/1956508725 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dfc0094f0 con 0x7f7e0c070fc0 2023-12-16T20:45:29.844 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.114+0000 7f7e0bfff700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e0c070fc0 0x7f7e0c1111c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc006d40 tx=0x7f7dfc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:29.844 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.114+0000 7f7e08ff9700 1 -- 172.21.15.52:0/1956508725 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dfc010960 con 0x7f7e0c070fc0 2023-12-16T20:45:29.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.114+0000 7f7e08ff9700 1 -- 172.21.15.52:0/1956508725 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7dfc01a430 con 0x7f7e0c070fc0 2023-12-16T20:45:29.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.114+0000 7f7e1275e700 1 -- 172.21.15.52:0/1956508725 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0c111900 con 0x7f7e0c070fc0 2023-12-16T20:45:29.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.114+0000 7f7e08ff9700 1 -- 172.21.15.52:0/1956508725 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dfc0245b0 con 0x7f7e0c070fc0 2023-12-16T20:45:29.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.114+0000 7f7e1275e700 1 -- 172.21.15.52:0/1956508725 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0c111d10 con 0x7f7e0c070fc0 2023-12-16T20:45:29.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.115+0000 7f7e08ff9700 1 -- 172.21.15.52:0/1956508725 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f7dfc02d3f0 con 0x7f7e0c070fc0 2023-12-16T20:45:29.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.115+0000 7f7e08ff9700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f7df8036640 0x7f7df8038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:29.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.115+0000 7f7e1275e700 1 -- 172.21.15.52:0/1956508725 --> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e0c04bb80 con 0x7f7df8036640 2023-12-16T20:45:29.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.115+0000 7f7e08ff9700 1 -- 172.21.15.52:0/1956508725 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7dfc04a210 con 0x7f7e0c070fc0 2023-12-16T20:45:29.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.115+0000 7f7e0b7fe700 1 -- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f7df8036640 msgr2=0x7f7df8038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/221719009 2023-12-16T20:45:29.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.115+0000 7f7e0b7fe700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f7df8036640 0x7f7df8038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T20:45:29.845 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.316+0000 7f7e0b7fe700 1 -- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f7df8036640 msgr2=0x7f7df8038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/221719009 2023-12-16T20:45:29.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.316+0000 7f7e0b7fe700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f7df8036640 0x7f7df8038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T20:45:29.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.717+0000 7f7e0b7fe700 1 -- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f7df8036640 msgr2=0x7f7df8038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/221719009 2023-12-16T20:45:29.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:23.717+0000 7f7e0b7fe700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f7df8036640 0x7f7df8038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-16T20:45:29.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:24.518+0000 7f7e0b7fe700 1 -- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f7df8036640 msgr2=0x7f7df8038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/221719009 2023-12-16T20:45:29.848 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:24.518+0000 7f7e0b7fe700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f7df8036640 0x7f7df8038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-16T20:45:29.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:26.119+0000 7f7e0b7fe700 1 -- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f7df8036640 msgr2=0x7f7df8038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/221719009 2023-12-16T20:45:29.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:26.119+0000 7f7e0b7fe700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f7df8036640 0x7f7df8038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-16T20:45:29.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:28.571+0000 7f7e08ff9700 1 -- 172.21.15.52:0/1956508725 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mgrmap(e 10) v1 ==== 43061+0+0 (secure 0 0 0) 0x7f7dfc01a9f0 con 0x7f7e0c070fc0 2023-12-16T20:45:29.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:28.572+0000 7f7e08ff9700 1 -- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f7df8036640 msgr2=0x7f7df8038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T20:45:29.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:28.572+0000 7f7e08ff9700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f7df8036640 0x7f7df8038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:29.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.573+0000 7f7e08ff9700 1 -- 172.21.15.52:0/1956508725 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7f7dfc018350 con 0x7f7e0c070fc0 2023-12-16T20:45:29.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.573+0000 7f7e08ff9700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f7df8039f40 0x7f7df803c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:29.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.573+0000 7f7e08ff9700 1 -- 172.21.15.52:0/1956508725 --> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e0c04bb80 con 0x7f7df8039f40 2023-12-16T20:45:29.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.574+0000 7f7e0b7fe700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f7df8039f40 0x7f7df803c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:29.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.574+0000 7f7e0b7fe700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f7df8039f40 0x7f7df803c320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7df40039d0 tx=0x7f7df40092f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:29.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.575+0000 7f7e08ff9700 1 -- 172.21.15.52:0/1956508725 <== mgr.14162 v2:172.21.15.52:6800/2799692744 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f7e0c04bb80 con 0x7f7df8039f40 2023-12-16T20:45:29.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.585+0000 7f7e1275e700 1 -- 172.21.15.52:0/1956508725 --> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f7e0c1062e0 con 0x7f7df8039f40 2023-12-16T20:45:29.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.585+0000 7f7e08ff9700 1 -- 172.21.15.52:0/1956508725 <== mgr.14162 v2:172.21.15.52:6800/2799692744 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f7e0c1062e0 con 0x7f7df8039f40 2023-12-16T20:45:29.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.586+0000 7f7e1275e700 1 -- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f7df8039f40 msgr2=0x7f7df803c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:29.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.586+0000 7f7e1275e700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f7df8039f40 0x7f7df803c320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7df40039d0 tx=0x7f7df40092f0 comp rx=0 tx=0).stop 2023-12-16T20:45:29.850 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.586+0000 7f7e1275e700 1 -- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e0c070fc0 msgr2=0x7f7e0c1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:29.850 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.586+0000 7f7e1275e700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e0c070fc0 0x7f7e0c1111c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc006d40 tx=0x7f7dfc009250 comp rx=0 tx=0).stop 2023-12-16T20:45:29.850 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.586+0000 7f7e1275e700 1 -- 172.21.15.52:0/1956508725 shutdown_connections 2023-12-16T20:45:29.850 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.586+0000 7f7e1275e700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f7df8039f40 0x7f7df803c320 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:29.850 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.586+0000 7f7e1275e700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e0c070fc0 0x7f7e0c1111c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:29.850 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.586+0000 7f7e1275e700 1 --2- 172.21.15.52:0/1956508725 >> [v2:172.21.15.52:6800/221719009,v1:172.21.15.52:6801/221719009] conn(0x7f7df8036640 0x7f7df8038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:29.850 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.586+0000 7f7e1275e700 1 -- 172.21.15.52:0/1956508725 >> 172.21.15.52:0/1956508725 conn(0x7f7e0c0fc970 msgr2=0x7f7e0c0fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:29.850 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.586+0000 7f7e1275e700 1 -- 172.21.15.52:0/1956508725 shutdown_connections 2023-12-16T20:45:29.850 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:29.586+0000 7f7e1275e700 1 -- 172.21.15.52:0/1956508725 wait complete. 2023-12-16T20:45:29.850 INFO:teuthology.orchestra.run.smithi052.stdout:mgr epoch 9 is available 2023-12-16T20:45:29.850 INFO:teuthology.orchestra.run.smithi052.stdout:Generating a dashboard self-signed certificate... 2023-12-16T20:45:30.642 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-12-16T20:45:30.642 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.110+0000 7f352a0cd700 1 Processor -- start 2023-12-16T20:45:30.642 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.110+0000 7f352a0cd700 1 -- start start 2023-12-16T20:45:30.642 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.110+0000 7f352a0cd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35241038f0 0x7f3524103d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:30.642 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.110+0000 7f352a0cd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3524104240 con 0x7f35241038f0 2023-12-16T20:45:30.642 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.111+0000 7f35237fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35241038f0 0x7f3524103d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:30.642 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.111+0000 7f35237fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35241038f0 0x7f3524103d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58138/0 (socket says 172.21.15.52:58138) 2023-12-16T20:45:30.643 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.111+0000 7f35237fe700 1 -- 172.21.15.52:0/3533974208 learned_addr learned my addr 172.21.15.52:0/3533974208 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:30.643 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.111+0000 7f35237fe700 1 -- 172.21.15.52:0/3533974208 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3524104380 con 0x7f35241038f0 2023-12-16T20:45:30.643 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.111+0000 7f35237fe700 1 --2- 172.21.15.52:0/3533974208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35241038f0 0x7f3524103d00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f35140098e0 tx=0x7f3514009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f664187674d83c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:30.643 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.112+0000 7f35227fc700 1 -- 172.21.15.52:0/3533974208 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f351400b860 con 0x7f35241038f0 2023-12-16T20:45:30.643 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.112+0000 7f35227fc700 1 -- 172.21.15.52:0/3533974208 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f351400be70 con 0x7f35241038f0 2023-12-16T20:45:30.643 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.112+0000 7f35227fc700 1 -- 172.21.15.52:0/3533974208 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3514010380 con 0x7f35241038f0 2023-12-16T20:45:30.643 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.112+0000 7f352a0cd700 1 -- 172.21.15.52:0/3533974208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35241038f0 msgr2=0x7f3524103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:30.643 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.112+0000 7f352a0cd700 1 --2- 172.21.15.52:0/3533974208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35241038f0 0x7f3524103d00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f35140098e0 tx=0x7f3514009250 comp rx=0 tx=0).stop 2023-12-16T20:45:30.643 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.113+0000 7f352a0cd700 1 -- 172.21.15.52:0/3533974208 shutdown_connections 2023-12-16T20:45:30.643 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.113+0000 7f352a0cd700 1 --2- 172.21.15.52:0/3533974208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35241038f0 0x7f3524103d00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:30.643 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.113+0000 7f352a0cd700 1 -- 172.21.15.52:0/3533974208 >> 172.21.15.52:0/3533974208 conn(0x7f35240fc0b0 msgr2=0x7f35240fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:30.643 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.113+0000 7f352a0cd700 1 -- 172.21.15.52:0/3533974208 shutdown_connections 2023-12-16T20:45:30.643 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.113+0000 7f352a0cd700 1 -- 172.21.15.52:0/3533974208 wait complete. 2023-12-16T20:45:30.643 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.113+0000 7f352a0cd700 1 Processor -- start 2023-12-16T20:45:30.643 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.114+0000 7f352a0cd700 1 -- start start 2023-12-16T20:45:30.644 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.114+0000 7f352a0cd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35241038f0 0x7f3524111140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:30.644 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.114+0000 7f352a0cd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3524111680 con 0x7f35241038f0 2023-12-16T20:45:30.644 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.114+0000 7f35237fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35241038f0 0x7f3524111140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:30.644 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.114+0000 7f35237fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35241038f0 0x7f3524111140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58150/0 (socket says 172.21.15.52:58150) 2023-12-16T20:45:30.644 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.114+0000 7f35237fe700 1 -- 172.21.15.52:0/1617310373 learned_addr learned my addr 172.21.15.52:0/1617310373 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:30.644 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.114+0000 7f35237fe700 1 -- 172.21.15.52:0/1617310373 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35140094f0 con 0x7f35241038f0 2023-12-16T20:45:30.644 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.114+0000 7f35237fe700 1 --2- 172.21.15.52:0/1617310373 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35241038f0 0x7f3524111140 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3514000c00 tx=0x7f3514009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:30.644 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.115+0000 7f350bfff700 1 -- 172.21.15.52:0/1617310373 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3514010800 con 0x7f35241038f0 2023-12-16T20:45:30.644 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.115+0000 7f350bfff700 1 -- 172.21.15.52:0/1617310373 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3514010e10 con 0x7f35241038f0 2023-12-16T20:45:30.644 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.115+0000 7f352a0cd700 1 -- 172.21.15.52:0/1617310373 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3524111880 con 0x7f35241038f0 2023-12-16T20:45:30.644 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.115+0000 7f352a0cd700 1 -- 172.21.15.52:0/1617310373 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3524111c90 con 0x7f35241038f0 2023-12-16T20:45:30.644 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.115+0000 7f350bfff700 1 -- 172.21.15.52:0/1617310373 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f351401ae90 con 0x7f35241038f0 2023-12-16T20:45:30.644 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.116+0000 7f350bfff700 1 -- 172.21.15.52:0/1617310373 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7f3514024500 con 0x7f35241038f0 2023-12-16T20:45:30.644 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.116+0000 7f350bfff700 1 --2- 172.21.15.52:0/1617310373 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f350c0365b0 0x7f350c038a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:30.644 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.116+0000 7f350bfff700 1 -- 172.21.15.52:0/1617310373 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f351404aa80 con 0x7f35241038f0 2023-12-16T20:45:30.645 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.116+0000 7f352a0cd700 1 -- 172.21.15.52:0/1617310373 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f352404bb80 con 0x7f35241038f0 2023-12-16T20:45:30.645 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.116+0000 7f3522ffd700 1 --2- 172.21.15.52:0/1617310373 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f350c0365b0 0x7f350c038a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:30.645 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.118+0000 7f3522ffd700 1 --2- 172.21.15.52:0/1617310373 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f350c0365b0 0x7f350c038a60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3518009990 tx=0x7f3518006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:30.645 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.125+0000 7f350bfff700 1 -- 172.21.15.52:0/1617310373 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f351402e440 con 0x7f35241038f0 2023-12-16T20:45:30.645 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.342+0000 7f352a0cd700 1 -- 172.21.15.52:0/1617310373 --> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f352410b080 con 0x7f350c0365b0 2023-12-16T20:45:30.645 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.394+0000 7f350bfff700 1 -- 172.21.15.52:0/1617310373 <== mgr.14162 v2:172.21.15.52:6800/2799692744 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f352410b080 con 0x7f350c0365b0 2023-12-16T20:45:30.645 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.396+0000 7f352a0cd700 1 -- 172.21.15.52:0/1617310373 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f350c0365b0 msgr2=0x7f350c038a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:30.645 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.396+0000 7f352a0cd700 1 --2- 172.21.15.52:0/1617310373 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f350c0365b0 0x7f350c038a60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3518009990 tx=0x7f3518006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:30.645 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.396+0000 7f352a0cd700 1 -- 172.21.15.52:0/1617310373 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35241038f0 msgr2=0x7f3524111140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:30.645 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.396+0000 7f352a0cd700 1 --2- 172.21.15.52:0/1617310373 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35241038f0 0x7f3524111140 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3514000c00 tx=0x7f3514009250 comp rx=0 tx=0).stop 2023-12-16T20:45:30.645 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.397+0000 7f352a0cd700 1 -- 172.21.15.52:0/1617310373 shutdown_connections 2023-12-16T20:45:30.645 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.397+0000 7f352a0cd700 1 --2- 172.21.15.52:0/1617310373 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f350c0365b0 0x7f350c038a60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:30.645 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.397+0000 7f352a0cd700 1 --2- 172.21.15.52:0/1617310373 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35241038f0 0x7f3524111140 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:30.646 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.397+0000 7f352a0cd700 1 -- 172.21.15.52:0/1617310373 >> 172.21.15.52:0/1617310373 conn(0x7f35240fc0b0 msgr2=0x7f35240fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:30.646 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.397+0000 7f352a0cd700 1 -- 172.21.15.52:0/1617310373 shutdown_connections 2023-12-16T20:45:30.646 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.397+0000 7f352a0cd700 1 -- 172.21.15.52:0/1617310373 wait complete. 2023-12-16T20:45:30.646 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial admin user... 2023-12-16T20:45:30.709 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:30 smithi052 ceph-mon[80761]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-16T20:45:30.709 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:30 smithi052 ceph-mon[80761]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-16T20:45:30.709 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:30 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:30.709 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:30 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:30.709 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:30 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:30.709 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:30 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:31.626 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$6gsIQm0zecqQLAb1VsPjN.FykbCSKIfQ/uoW83dv9cx2ICuYQScfy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1702759531, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-12-16T20:45:31.627 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.900+0000 7f1a4853c700 1 Processor -- start 2023-12-16T20:45:31.627 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.900+0000 7f1a4853c700 1 -- start start 2023-12-16T20:45:31.627 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.900+0000 7f1a4853c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a40078dc0 0x7f1a400771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:31.627 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.900+0000 7f1a4853c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a40077710 con 0x7f1a40078dc0 2023-12-16T20:45:31.627 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.900+0000 7f1a462d8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a40078dc0 0x7f1a400771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:31.627 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.900+0000 7f1a462d8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a40078dc0 0x7f1a400771d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58156/0 (socket says 172.21.15.52:58156) 2023-12-16T20:45:31.627 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.900+0000 7f1a462d8700 1 -- 172.21.15.52:0/853872101 learned_addr learned my addr 172.21.15.52:0/853872101 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:31.627 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.901+0000 7f1a462d8700 1 -- 172.21.15.52:0/853872101 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a40077850 con 0x7f1a40078dc0 2023-12-16T20:45:31.627 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.901+0000 7f1a462d8700 1 --2- 172.21.15.52:0/853872101 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a40078dc0 0x7f1a400771d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c0098e0 tx=0x7f1a3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3ff71de3f38308 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:31.627 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.901+0000 7f1a452d6700 1 -- 172.21.15.52:0/853872101 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a3c00b860 con 0x7f1a40078dc0 2023-12-16T20:45:31.627 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.901+0000 7f1a452d6700 1 -- 172.21.15.52:0/853872101 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a3c00be70 con 0x7f1a40078dc0 2023-12-16T20:45:31.627 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.901+0000 7f1a452d6700 1 -- 172.21.15.52:0/853872101 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a3c010380 con 0x7f1a40078dc0 2023-12-16T20:45:31.627 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.901+0000 7f1a4853c700 1 -- 172.21.15.52:0/853872101 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a40078dc0 msgr2=0x7f1a400771d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:31.628 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.901+0000 7f1a4853c700 1 --2- 172.21.15.52:0/853872101 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a40078dc0 0x7f1a400771d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c0098e0 tx=0x7f1a3c009250 comp rx=0 tx=0).stop 2023-12-16T20:45:31.628 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.902+0000 7f1a4853c700 1 -- 172.21.15.52:0/853872101 shutdown_connections 2023-12-16T20:45:31.628 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.902+0000 7f1a4853c700 1 --2- 172.21.15.52:0/853872101 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a40078dc0 0x7f1a400771d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:31.628 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.902+0000 7f1a4853c700 1 -- 172.21.15.52:0/853872101 >> 172.21.15.52:0/853872101 conn(0x7f1a400fda40 msgr2=0x7f1a400ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:31.628 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.902+0000 7f1a4853c700 1 -- 172.21.15.52:0/853872101 shutdown_connections 2023-12-16T20:45:31.628 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.902+0000 7f1a4853c700 1 -- 172.21.15.52:0/853872101 wait complete. 2023-12-16T20:45:31.628 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.902+0000 7f1a4853c700 1 Processor -- start 2023-12-16T20:45:31.628 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.902+0000 7f1a4853c700 1 -- start start 2023-12-16T20:45:31.628 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.902+0000 7f1a4853c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a40078dc0 0x7f1a40113380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:31.628 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.902+0000 7f1a4853c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a401138c0 con 0x7f1a40078dc0 2023-12-16T20:45:31.628 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.902+0000 7f1a462d8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a40078dc0 0x7f1a40113380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:31.628 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.903+0000 7f1a462d8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a40078dc0 0x7f1a40113380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58158/0 (socket says 172.21.15.52:58158) 2023-12-16T20:45:31.628 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.903+0000 7f1a462d8700 1 -- 172.21.15.52:0/776500397 learned_addr learned my addr 172.21.15.52:0/776500397 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:31.628 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.903+0000 7f1a462d8700 1 -- 172.21.15.52:0/776500397 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3c0094f0 con 0x7f1a40078dc0 2023-12-16T20:45:31.629 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.903+0000 7f1a462d8700 1 --2- 172.21.15.52:0/776500397 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a40078dc0 0x7f1a40113380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c0094c0 tx=0x7f1a3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:31.629 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.903+0000 7f1a36ffd700 1 -- 172.21.15.52:0/776500397 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a3c010850 con 0x7f1a40078dc0 2023-12-16T20:45:31.629 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.903+0000 7f1a36ffd700 1 -- 172.21.15.52:0/776500397 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a3c010e60 con 0x7f1a40078dc0 2023-12-16T20:45:31.629 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.903+0000 7f1a36ffd700 1 -- 172.21.15.52:0/776500397 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a3c0243f0 con 0x7f1a40078dc0 2023-12-16T20:45:31.629 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.903+0000 7f1a4853c700 1 -- 172.21.15.52:0/776500397 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a40113ac0 con 0x7f1a40078dc0 2023-12-16T20:45:31.629 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.903+0000 7f1a4853c700 1 -- 172.21.15.52:0/776500397 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a40113ed0 con 0x7f1a40078dc0 2023-12-16T20:45:31.629 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.904+0000 7f1a36ffd700 1 -- 172.21.15.52:0/776500397 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7f1a3c01a430 con 0x7f1a40078dc0 2023-12-16T20:45:31.629 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.904+0000 7f1a4853c700 1 -- 172.21.15.52:0/776500397 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a40108620 con 0x7f1a40078dc0 2023-12-16T20:45:31.629 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.904+0000 7f1a36ffd700 1 --2- 172.21.15.52:0/776500397 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f1a2c036570 0x7f1a2c038a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:31.629 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.904+0000 7f1a36ffd700 1 -- 172.21.15.52:0/776500397 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1a3c04a500 con 0x7f1a40078dc0 2023-12-16T20:45:31.629 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.905+0000 7f1a45ad7700 1 --2- 172.21.15.52:0/776500397 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f1a2c036570 0x7f1a2c038a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:31.629 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.905+0000 7f1a45ad7700 1 --2- 172.21.15.52:0/776500397 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f1a2c036570 0x7f1a2c038a20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009990 tx=0x7f1a30006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:31.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:30.909+0000 7f1a36ffd700 1 -- 172.21.15.52:0/776500397 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1a3c018b40 con 0x7f1a40078dc0 2023-12-16T20:45:31.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.137+0000 7f1a4853c700 1 -- 172.21.15.52:0/776500397 --> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1a4004bb80 con 0x7f1a2c036570 2023-12-16T20:45:31.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.383+0000 7f1a36ffd700 1 -- 172.21.15.52:0/776500397 <== mgr.14162 v2:172.21.15.52:6800/2799692744 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f1a4004bb80 con 0x7f1a2c036570 2023-12-16T20:45:31.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.385+0000 7f1a4853c700 1 -- 172.21.15.52:0/776500397 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f1a2c036570 msgr2=0x7f1a2c038a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:31.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.385+0000 7f1a4853c700 1 --2- 172.21.15.52:0/776500397 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f1a2c036570 0x7f1a2c038a20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009990 tx=0x7f1a30006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:31.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.385+0000 7f1a4853c700 1 -- 172.21.15.52:0/776500397 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a40078dc0 msgr2=0x7f1a40113380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:31.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.385+0000 7f1a4853c700 1 --2- 172.21.15.52:0/776500397 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a40078dc0 0x7f1a40113380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c0094c0 tx=0x7f1a3c009250 comp rx=0 tx=0).stop 2023-12-16T20:45:31.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.385+0000 7f1a4853c700 1 -- 172.21.15.52:0/776500397 shutdown_connections 2023-12-16T20:45:31.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.385+0000 7f1a4853c700 1 --2- 172.21.15.52:0/776500397 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f1a2c036570 0x7f1a2c038a20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:31.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.385+0000 7f1a4853c700 1 --2- 172.21.15.52:0/776500397 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a40078dc0 0x7f1a40113380 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:31.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.386+0000 7f1a4853c700 1 -- 172.21.15.52:0/776500397 >> 172.21.15.52:0/776500397 conn(0x7f1a400fda40 msgr2=0x7f1a400ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:31.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.386+0000 7f1a4853c700 1 -- 172.21.15.52:0/776500397 shutdown_connections 2023-12-16T20:45:31.630 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.386+0000 7f1a4853c700 1 -- 172.21.15.52:0/776500397 wait complete. 2023-12-16T20:45:31.630 INFO:teuthology.orchestra.run.smithi052.stdout:Fetching dashboard port number... 2023-12-16T20:45:31.869 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:31 smithi052 ceph-mon[80761]: [16/Dec/2023:20:45:30] ENGINE Bus STARTING 2023-12-16T20:45:31.869 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:31 smithi052 ceph-mon[80761]: [16/Dec/2023:20:45:30] ENGINE Serving on https://172.21.15.52:7150 2023-12-16T20:45:31.869 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:31 smithi052 ceph-mon[80761]: [16/Dec/2023:20:45:30] ENGINE Bus STARTED 2023-12-16T20:45:31.869 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:31 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:31.869 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:31 smithi052 ceph-mon[80761]: mgrmap e12: smithi052.nmkcbs(active, since 2s) 2023-12-16T20:45:32.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 8443 2023-12-16T20:45:32.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.868+0000 7f6851e2a700 1 Processor -- start 2023-12-16T20:45:32.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.868+0000 7f6851e2a700 1 -- start start 2023-12-16T20:45:32.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.868+0000 7f6851e2a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f684c1039c0 0x7f684c103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:32.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.868+0000 7f6851e2a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c104310 con 0x7f684c1039c0 2023-12-16T20:45:32.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.868+0000 7f684b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f684c1039c0 0x7f684c103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:32.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.868+0000 7f684b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f684c1039c0 0x7f684c103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58168/0 (socket says 172.21.15.52:58168) 2023-12-16T20:45:32.335 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.868+0000 7f684b7fe700 1 -- 172.21.15.52:0/742279545 learned_addr learned my addr 172.21.15.52:0/742279545 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:32.335 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.868+0000 7f684b7fe700 1 -- 172.21.15.52:0/742279545 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c101fb0 con 0x7f684c1039c0 2023-12-16T20:45:32.335 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.869+0000 7f684b7fe700 1 --2- 172.21.15.52:0/742279545 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f684c1039c0 0x7f684c103dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f683c0098e0 tx=0x7f683c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe973d9e7c76cab server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:32.335 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.869+0000 7f684a7fc700 1 -- 172.21.15.52:0/742279545 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f683c00b860 con 0x7f684c1039c0 2023-12-16T20:45:32.335 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.869+0000 7f684a7fc700 1 -- 172.21.15.52:0/742279545 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f683c00be70 con 0x7f684c1039c0 2023-12-16T20:45:32.335 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.869+0000 7f684a7fc700 1 -- 172.21.15.52:0/742279545 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f683c010380 con 0x7f684c1039c0 2023-12-16T20:45:32.336 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.869+0000 7f6851e2a700 1 -- 172.21.15.52:0/742279545 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f684c1039c0 msgr2=0x7f684c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:32.336 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.869+0000 7f6851e2a700 1 --2- 172.21.15.52:0/742279545 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f684c1039c0 0x7f684c103dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f683c0098e0 tx=0x7f683c009250 comp rx=0 tx=0).stop 2023-12-16T20:45:32.336 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.870+0000 7f6851e2a700 1 -- 172.21.15.52:0/742279545 shutdown_connections 2023-12-16T20:45:32.336 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.870+0000 7f6851e2a700 1 --2- 172.21.15.52:0/742279545 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f684c1039c0 0x7f684c103dd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:32.336 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.870+0000 7f6851e2a700 1 -- 172.21.15.52:0/742279545 >> 172.21.15.52:0/742279545 conn(0x7f684c0fda40 msgr2=0x7f684c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:32.336 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.870+0000 7f6851e2a700 1 -- 172.21.15.52:0/742279545 shutdown_connections 2023-12-16T20:45:32.336 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.870+0000 7f6851e2a700 1 -- 172.21.15.52:0/742279545 wait complete. 2023-12-16T20:45:32.336 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.870+0000 7f6851e2a700 1 Processor -- start 2023-12-16T20:45:32.336 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.870+0000 7f6851e2a700 1 -- start start 2023-12-16T20:45:32.336 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.871+0000 7f6851e2a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f684c1039c0 0x7f684c111130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:32.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.871+0000 7f6851e2a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c111670 con 0x7f684c1039c0 2023-12-16T20:45:32.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.871+0000 7f684b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f684c1039c0 0x7f684c111130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:32.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.871+0000 7f684b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f684c1039c0 0x7f684c111130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58182/0 (socket says 172.21.15.52:58182) 2023-12-16T20:45:32.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.871+0000 7f684b7fe700 1 -- 172.21.15.52:0/482916970 learned_addr learned my addr 172.21.15.52:0/482916970 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:32.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.871+0000 7f684b7fe700 1 -- 172.21.15.52:0/482916970 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683c0094f0 con 0x7f684c1039c0 2023-12-16T20:45:32.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.871+0000 7f684b7fe700 1 --2- 172.21.15.52:0/482916970 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f684c1039c0 0x7f684c111130 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f683c00bac0 tx=0x7f683c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:32.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.871+0000 7f6833fff700 1 -- 172.21.15.52:0/482916970 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f683c0109b0 con 0x7f684c1039c0 2023-12-16T20:45:32.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.871+0000 7f6833fff700 1 -- 172.21.15.52:0/482916970 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f683c01a430 con 0x7f684c1039c0 2023-12-16T20:45:32.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.871+0000 7f6833fff700 1 -- 172.21.15.52:0/482916970 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f683c0245b0 con 0x7f684c1039c0 2023-12-16T20:45:32.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.871+0000 7f6851e2a700 1 -- 172.21.15.52:0/482916970 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f684c111870 con 0x7f684c1039c0 2023-12-16T20:45:32.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.872+0000 7f6851e2a700 1 -- 172.21.15.52:0/482916970 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684c111c80 con 0x7f684c1039c0 2023-12-16T20:45:32.338 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.872+0000 7f6833fff700 1 -- 172.21.15.52:0/482916970 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f683c01a5a0 con 0x7f684c1039c0 2023-12-16T20:45:32.338 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.873+0000 7f6833fff700 1 --2- 172.21.15.52:0/482916970 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f6834036690 0x7f6834038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:32.338 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.873+0000 7f6851e2a700 1 -- 172.21.15.52:0/482916970 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f684c102d30 con 0x7f684c1039c0 2023-12-16T20:45:32.338 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.873+0000 7f6833fff700 1 -- 172.21.15.52:0/482916970 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f683c04a680 con 0x7f684c1039c0 2023-12-16T20:45:32.338 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.873+0000 7f684affd700 1 --2- 172.21.15.52:0/482916970 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f6834036690 0x7f6834038b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:32.338 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.873+0000 7f684affd700 1 --2- 172.21.15.52:0/482916970 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f6834036690 0x7f6834038b40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6840009990 tx=0x7f6840006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:32.338 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:31.877+0000 7f6833fff700 1 -- 172.21.15.52:0/482916970 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f683c010b10 con 0x7f684c1039c0 2023-12-16T20:45:32.338 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.072+0000 7f6851e2a700 1 -- 172.21.15.52:0/482916970 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f684c04bb80 con 0x7f684c1039c0 2023-12-16T20:45:32.338 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.072+0000 7f6833fff700 1 -- 172.21.15.52:0/482916970 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f683c018770 con 0x7f684c1039c0 2023-12-16T20:45:32.338 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.074+0000 7f6851e2a700 1 -- 172.21.15.52:0/482916970 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f6834036690 msgr2=0x7f6834038b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:32.339 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.074+0000 7f6851e2a700 1 --2- 172.21.15.52:0/482916970 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f6834036690 0x7f6834038b40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6840009990 tx=0x7f6840006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:32.339 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.074+0000 7f6851e2a700 1 -- 172.21.15.52:0/482916970 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f684c1039c0 msgr2=0x7f684c111130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:32.339 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.074+0000 7f6851e2a700 1 --2- 172.21.15.52:0/482916970 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f684c1039c0 0x7f684c111130 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f683c00bac0 tx=0x7f683c009250 comp rx=0 tx=0).stop 2023-12-16T20:45:32.339 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.074+0000 7f6851e2a700 1 -- 172.21.15.52:0/482916970 shutdown_connections 2023-12-16T20:45:32.339 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.074+0000 7f6851e2a700 1 --2- 172.21.15.52:0/482916970 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f6834036690 0x7f6834038b40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:32.339 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.074+0000 7f6851e2a700 1 --2- 172.21.15.52:0/482916970 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f684c1039c0 0x7f684c111130 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:32.339 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.074+0000 7f6851e2a700 1 -- 172.21.15.52:0/482916970 >> 172.21.15.52:0/482916970 conn(0x7f684c0fda40 msgr2=0x7f684c0ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:32.339 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.074+0000 7f6851e2a700 1 -- 172.21.15.52:0/482916970 shutdown_connections 2023-12-16T20:45:32.339 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.074+0000 7f6851e2a700 1 -- 172.21.15.52:0/482916970 wait complete. 2023-12-16T20:45:32.340 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-16T20:45:32.340 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2023-12-16T20:45:32.348 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-16T20:45:32.348 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2023-12-16T20:45:32.348 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2023-12-16T20:45:32.348 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-12-16T20:45:32.349 INFO:teuthology.orchestra.run.smithi052.stdout:Ceph Dashboard is now available at: 2023-12-16T20:45:32.350 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:45:32.350 INFO:teuthology.orchestra.run.smithi052.stdout: URL: https://smithi052.front.sepia.ceph.com:8443/ 2023-12-16T20:45:32.350 INFO:teuthology.orchestra.run.smithi052.stdout: User: admin 2023-12-16T20:45:32.350 INFO:teuthology.orchestra.run.smithi052.stdout: Password: qv7jkfl6ag 2023-12-16T20:45:32.350 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:45:32.350 INFO:teuthology.orchestra.run.smithi052.stdout:Saving cluster configuration to /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config directory 2023-12-16T20:45:32.350 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling autotune for osd_memory_target 2023-12-16T20:45:32.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:32 smithi052 ceph-mon[80761]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:32.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:32 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/482916970' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-12-16T20:45:33.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.592+0000 7f5e1dba9700 1 Processor -- start 2023-12-16T20:45:33.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.592+0000 7f5e1dba9700 1 -- start start 2023-12-16T20:45:33.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.592+0000 7f5e1dba9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5e181039c0 0x7f5e18103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:33.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.592+0000 7f5e1dba9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e18104310 con 0x7f5e181039c0 2023-12-16T20:45:33.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.592+0000 7f5e177fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5e181039c0 0x7f5e18103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:33.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.592+0000 7f5e177fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5e181039c0 0x7f5e18103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47482/0 (socket says 172.21.15.52:47482) 2023-12-16T20:45:33.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.592+0000 7f5e177fe700 1 -- 172.21.15.52:0/671341602 learned_addr learned my addr 172.21.15.52:0/671341602 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:33.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.592+0000 7f5e177fe700 1 -- 172.21.15.52:0/671341602 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e18101fb0 con 0x7f5e181039c0 2023-12-16T20:45:33.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.593+0000 7f5e177fe700 1 --2- 172.21.15.52:0/671341602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5e181039c0 0x7f5e18103dd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5e080098e0 tx=0x7f5e08009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ad7f8bc46e566 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:33.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.593+0000 7f5e167fc700 1 -- 172.21.15.52:0/671341602 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e0800b860 con 0x7f5e181039c0 2023-12-16T20:45:33.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.593+0000 7f5e167fc700 1 -- 172.21.15.52:0/671341602 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e0800be70 con 0x7f5e181039c0 2023-12-16T20:45:33.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.593+0000 7f5e167fc700 1 -- 172.21.15.52:0/671341602 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e08010380 con 0x7f5e181039c0 2023-12-16T20:45:33.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.593+0000 7f5e1dba9700 1 -- 172.21.15.52:0/671341602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5e181039c0 msgr2=0x7f5e18103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:33.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.593+0000 7f5e1dba9700 1 --2- 172.21.15.52:0/671341602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5e181039c0 0x7f5e18103dd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5e080098e0 tx=0x7f5e08009250 comp rx=0 tx=0).stop 2023-12-16T20:45:33.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.593+0000 7f5e1dba9700 1 -- 172.21.15.52:0/671341602 shutdown_connections 2023-12-16T20:45:33.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.593+0000 7f5e1dba9700 1 --2- 172.21.15.52:0/671341602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5e181039c0 0x7f5e18103dd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:33.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.593+0000 7f5e1dba9700 1 -- 172.21.15.52:0/671341602 >> 172.21.15.52:0/671341602 conn(0x7f5e180fda40 msgr2=0x7f5e180ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:33.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.594+0000 7f5e1dba9700 1 -- 172.21.15.52:0/671341602 shutdown_connections 2023-12-16T20:45:33.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.594+0000 7f5e1dba9700 1 -- 172.21.15.52:0/671341602 wait complete. 2023-12-16T20:45:33.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.594+0000 7f5e1dba9700 1 Processor -- start 2023-12-16T20:45:33.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.594+0000 7f5e1dba9700 1 -- start start 2023-12-16T20:45:33.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.594+0000 7f5e1dba9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5e181039c0 0x7f5e18078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:33.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.594+0000 7f5e1dba9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e18077190 con 0x7f5e181039c0 2023-12-16T20:45:33.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.595+0000 7f5e177fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5e181039c0 0x7f5e18078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:33.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.595+0000 7f5e177fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5e181039c0 0x7f5e18078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47484/0 (socket says 172.21.15.52:47484) 2023-12-16T20:45:33.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.595+0000 7f5e177fe700 1 -- 172.21.15.52:0/1560220833 learned_addr learned my addr 172.21.15.52:0/1560220833 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:33.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.595+0000 7f5e177fe700 1 -- 172.21.15.52:0/1560220833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e080094f0 con 0x7f5e181039c0 2023-12-16T20:45:33.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.595+0000 7f5e177fe700 1 --2- 172.21.15.52:0/1560220833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5e181039c0 0x7f5e18078d10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5e08003080 tx=0x7f5e08009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:33.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.595+0000 7f5dfffff700 1 -- 172.21.15.52:0/1560220833 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e08010800 con 0x7f5e181039c0 2023-12-16T20:45:33.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.595+0000 7f5e1dba9700 1 -- 172.21.15.52:0/1560220833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e18077390 con 0x7f5e181039c0 2023-12-16T20:45:33.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.595+0000 7f5dfffff700 1 -- 172.21.15.52:0/1560220833 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e08010e10 con 0x7f5e181039c0 2023-12-16T20:45:33.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.595+0000 7f5dfffff700 1 -- 172.21.15.52:0/1560220833 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e08024ea0 con 0x7f5e181039c0 2023-12-16T20:45:33.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.595+0000 7f5e1dba9700 1 -- 172.21.15.52:0/1560220833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e180777a0 con 0x7f5e181039c0 2023-12-16T20:45:33.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.596+0000 7f5dfffff700 1 -- 172.21.15.52:0/1560220833 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f5e0801a5b0 con 0x7f5e181039c0 2023-12-16T20:45:33.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.596+0000 7f5dfffff700 1 --2- 172.21.15.52:0/1560220833 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f5e00036640 0x7f5e00038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:33.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.596+0000 7f5dfffff700 1 -- 172.21.15.52:0/1560220833 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5e0804a5b0 con 0x7f5e181039c0 2023-12-16T20:45:33.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.596+0000 7f5e1dba9700 1 -- 172.21.15.52:0/1560220833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e1805f270 con 0x7f5e181039c0 2023-12-16T20:45:33.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.596+0000 7f5e16ffd700 1 --2- 172.21.15.52:0/1560220833 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f5e00036640 0x7f5e00038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:33.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.597+0000 7f5e16ffd700 1 --2- 172.21.15.52:0/1560220833 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f5e00036640 0x7f5e00038af0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009990 tx=0x7f5e0c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:33.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.601+0000 7f5dfffff700 1 -- 172.21.15.52:0/1560220833 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5e0802d3f0 con 0x7f5e181039c0 2023-12-16T20:45:33.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.799+0000 7f5e1dba9700 1 -- 172.21.15.52:0/1560220833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f5e18102d30 con 0x7f5e181039c0 2023-12-16T20:45:33.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.800+0000 7f5dfffff700 1 -- 172.21.15.52:0/1560220833 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f5e08048020 con 0x7f5e181039c0 2023-12-16T20:45:33.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.801+0000 7f5e1dba9700 1 -- 172.21.15.52:0/1560220833 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f5e00036640 msgr2=0x7f5e00038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:33.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.801+0000 7f5e1dba9700 1 --2- 172.21.15.52:0/1560220833 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f5e00036640 0x7f5e00038af0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009990 tx=0x7f5e0c006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:33.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.802+0000 7f5e1dba9700 1 -- 172.21.15.52:0/1560220833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5e181039c0 msgr2=0x7f5e18078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:33.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.802+0000 7f5e1dba9700 1 --2- 172.21.15.52:0/1560220833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5e181039c0 0x7f5e18078d10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5e08003080 tx=0x7f5e08009250 comp rx=0 tx=0).stop 2023-12-16T20:45:33.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.802+0000 7f5e1dba9700 1 -- 172.21.15.52:0/1560220833 shutdown_connections 2023-12-16T20:45:33.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.802+0000 7f5e1dba9700 1 --2- 172.21.15.52:0/1560220833 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f5e00036640 0x7f5e00038af0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:33.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.802+0000 7f5e1dba9700 1 --2- 172.21.15.52:0/1560220833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5e181039c0 0x7f5e18078d10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:33.045 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.802+0000 7f5e1dba9700 1 -- 172.21.15.52:0/1560220833 >> 172.21.15.52:0/1560220833 conn(0x7f5e180fda40 msgr2=0x7f5e180ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:33.046 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.802+0000 7f5e1dba9700 1 -- 172.21.15.52:0/1560220833 shutdown_connections 2023-12-16T20:45:33.046 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:32.802+0000 7f5e1dba9700 1 -- 172.21.15.52:0/1560220833 wait complete. 2023-12-16T20:45:33.867 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.275+0000 7f35ad43c700 1 Processor -- start 2023-12-16T20:45:33.867 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.275+0000 7f35ad43c700 1 -- start start 2023-12-16T20:45:33.867 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.275+0000 7f35ad43c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35a81011d0 0x7f35a81015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:33.867 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.275+0000 7f35ad43c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a8101b20 con 0x7f35a81011d0 2023-12-16T20:45:33.867 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.275+0000 7f35a6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35a81011d0 0x7f35a81015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:33.868 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.275+0000 7f35a6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35a81011d0 0x7f35a81015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47494/0 (socket says 172.21.15.52:47494) 2023-12-16T20:45:33.868 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.275+0000 7f35a6ffd700 1 -- 172.21.15.52:0/1834549212 learned_addr learned my addr 172.21.15.52:0/1834549212 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:33.868 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.275+0000 7f35a6ffd700 1 -- 172.21.15.52:0/1834549212 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a8101c60 con 0x7f35a81011d0 2023-12-16T20:45:33.868 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.276+0000 7f35a6ffd700 1 --2- 172.21.15.52:0/1834549212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35a81011d0 0x7f35a81015e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f359800bb80 tx=0x7f359800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee00110a54c112cd server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:33.868 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.276+0000 7f35a5ffb700 1 -- 172.21.15.52:0/1834549212 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f359800dd00 con 0x7f35a81011d0 2023-12-16T20:45:33.868 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.276+0000 7f35a5ffb700 1 -- 172.21.15.52:0/1834549212 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f359800de60 con 0x7f35a81011d0 2023-12-16T20:45:33.868 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.276+0000 7f35a5ffb700 1 -- 172.21.15.52:0/1834549212 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35980188d0 con 0x7f35a81011d0 2023-12-16T20:45:33.868 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.276+0000 7f35ad43c700 1 -- 172.21.15.52:0/1834549212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35a81011d0 msgr2=0x7f35a81015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:33.868 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.276+0000 7f35ad43c700 1 --2- 172.21.15.52:0/1834549212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35a81011d0 0x7f35a81015e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f359800bb80 tx=0x7f359800b4f0 comp rx=0 tx=0).stop 2023-12-16T20:45:33.868 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.276+0000 7f35ad43c700 1 -- 172.21.15.52:0/1834549212 shutdown_connections 2023-12-16T20:45:33.868 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.276+0000 7f35ad43c700 1 --2- 172.21.15.52:0/1834549212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35a81011d0 0x7f35a81015e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:33.868 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.276+0000 7f35ad43c700 1 -- 172.21.15.52:0/1834549212 >> 172.21.15.52:0/1834549212 conn(0x7f35a80fc950 msgr2=0x7f35a80fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:33.868 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.276+0000 7f35ad43c700 1 -- 172.21.15.52:0/1834549212 shutdown_connections 2023-12-16T20:45:33.868 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.276+0000 7f35ad43c700 1 -- 172.21.15.52:0/1834549212 wait complete. 2023-12-16T20:45:33.868 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.277+0000 7f35ad43c700 1 Processor -- start 2023-12-16T20:45:33.869 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.277+0000 7f35ad43c700 1 -- start start 2023-12-16T20:45:33.869 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.277+0000 7f35ad43c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35a81011d0 0x7f35a810ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:33.869 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.277+0000 7f35ad43c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a810f470 con 0x7f35a81011d0 2023-12-16T20:45:33.869 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.277+0000 7f35a6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35a81011d0 0x7f35a810ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:33.869 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.277+0000 7f35a6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35a81011d0 0x7f35a810ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47508/0 (socket says 172.21.15.52:47508) 2023-12-16T20:45:33.869 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.277+0000 7f35a6ffd700 1 -- 172.21.15.52:0/1846516168 learned_addr learned my addr 172.21.15.52:0/1846516168 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:33.869 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.278+0000 7f35a6ffd700 1 -- 172.21.15.52:0/1846516168 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359800b790 con 0x7f35a81011d0 2023-12-16T20:45:33.869 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.278+0000 7f35a6ffd700 1 --2- 172.21.15.52:0/1846516168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35a81011d0 0x7f35a810ef30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f35980184f0 tx=0x7f359800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:33.869 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.278+0000 7f358f7fe700 1 -- 172.21.15.52:0/1846516168 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3598010bd0 con 0x7f35a81011d0 2023-12-16T20:45:33.869 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.278+0000 7f358f7fe700 1 -- 172.21.15.52:0/1846516168 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f359801d050 con 0x7f35a81011d0 2023-12-16T20:45:33.869 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.278+0000 7f35ad43c700 1 -- 172.21.15.52:0/1846516168 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a810f670 con 0x7f35a81011d0 2023-12-16T20:45:33.869 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.278+0000 7f358f7fe700 1 -- 172.21.15.52:0/1846516168 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3598022590 con 0x7f35a81011d0 2023-12-16T20:45:33.869 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.278+0000 7f35ad43c700 1 -- 172.21.15.52:0/1846516168 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a810fa80 con 0x7f35a81011d0 2023-12-16T20:45:33.869 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.279+0000 7f358f7fe700 1 -- 172.21.15.52:0/1846516168 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f3598022800 con 0x7f35a81011d0 2023-12-16T20:45:33.870 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.279+0000 7f358f7fe700 1 --2- 172.21.15.52:0/1846516168 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f3590036690 0x7f3590038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:33.870 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.279+0000 7f358f7fe700 1 -- 172.21.15.52:0/1846516168 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3598014070 con 0x7f35a81011d0 2023-12-16T20:45:33.870 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.279+0000 7f35a67fc700 1 --2- 172.21.15.52:0/1846516168 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f3590036690 0x7f3590038b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:33.870 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.279+0000 7f35ad43c700 1 -- 172.21.15.52:0/1846516168 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a804bb80 con 0x7f35a81011d0 2023-12-16T20:45:33.870 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.280+0000 7f35a67fc700 1 --2- 172.21.15.52:0/1846516168 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f3590036690 0x7f3590038b40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f359c009990 tx=0x7f359c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:33.870 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.284+0000 7f358f7fe700 1 -- 172.21.15.52:0/1846516168 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3598022b50 con 0x7f35a81011d0 2023-12-16T20:45:33.870 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.593+0000 7f35ad43c700 1 -- 172.21.15.52:0/1846516168 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f35a8102fb0 con 0x7f35a81011d0 2023-12-16T20:45:33.870 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.595+0000 7f358f7fe700 1 -- 172.21.15.52:0/1846516168 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f359802be90 con 0x7f35a81011d0 2023-12-16T20:45:33.870 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.598+0000 7f35ad43c700 1 -- 172.21.15.52:0/1846516168 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f3590036690 msgr2=0x7f3590038b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:33.870 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.598+0000 7f35ad43c700 1 --2- 172.21.15.52:0/1846516168 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f3590036690 0x7f3590038b40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f359c009990 tx=0x7f359c006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:33.870 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.598+0000 7f35ad43c700 1 -- 172.21.15.52:0/1846516168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35a81011d0 msgr2=0x7f35a810ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:33.870 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.598+0000 7f35ad43c700 1 --2- 172.21.15.52:0/1846516168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35a81011d0 0x7f35a810ef30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f35980184f0 tx=0x7f359800b4f0 comp rx=0 tx=0).stop 2023-12-16T20:45:33.871 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.598+0000 7f35ad43c700 1 -- 172.21.15.52:0/1846516168 shutdown_connections 2023-12-16T20:45:33.871 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.598+0000 7f35ad43c700 1 --2- 172.21.15.52:0/1846516168 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f3590036690 0x7f3590038b40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:33.871 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.598+0000 7f35ad43c700 1 --2- 172.21.15.52:0/1846516168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35a81011d0 0x7f35a810ef30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:33.871 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.598+0000 7f35ad43c700 1 -- 172.21.15.52:0/1846516168 >> 172.21.15.52:0/1846516168 conn(0x7f35a80fc950 msgr2=0x7f35a8105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:33.871 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.598+0000 7f35ad43c700 1 -- 172.21.15.52:0/1846516168 shutdown_connections 2023-12-16T20:45:33.871 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2023-12-16T20:45:33.598+0000 7f35ad43c700 1 -- 172.21.15.52:0/1846516168 wait complete. 2023-12-16T20:45:33.871 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-12-16T20:45:33.871 INFO:teuthology.orchestra.run.smithi052.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-12-16T20:45:33.871 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:45:33.871 INFO:teuthology.orchestra.run.smithi052.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-12-16T20:45:33.871 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:45:33.871 INFO:teuthology.orchestra.run.smithi052.stdout:Or, if you are only running a single cluster on this host: 2023-12-16T20:45:33.871 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:45:33.871 INFO:teuthology.orchestra.run.smithi052.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-12-16T20:45:33.871 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:45:33.872 INFO:teuthology.orchestra.run.smithi052.stdout:Please consider enabling telemetry to help improve Ceph: 2023-12-16T20:45:33.872 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:45:33.872 INFO:teuthology.orchestra.run.smithi052.stdout: ceph telemetry on 2023-12-16T20:45:33.872 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:45:33.872 INFO:teuthology.orchestra.run.smithi052.stdout:For more information see: 2023-12-16T20:45:33.872 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:45:33.872 INFO:teuthology.orchestra.run.smithi052.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2023-12-16T20:45:33.872 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:45:33.872 INFO:teuthology.orchestra.run.smithi052.stdout:Bootstrap complete. 2023-12-16T20:45:33.908 INFO:tasks.cephadm:Fetching config... 2023-12-16T20:45:33.908 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:45:33.908 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-12-16T20:45:33.964 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-12-16T20:45:33.964 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:45:33.965 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-12-16T20:45:34.023 INFO:tasks.cephadm:Fetching mon keyring... 2023-12-16T20:45:34.023 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:45:34.023 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/keyring of=/dev/stdout 2023-12-16T20:45:34.097 INFO:tasks.cephadm:Fetching pub ssh key... 2023-12-16T20:45:34.097 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:45:34.097 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-12-16T20:45:34.152 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-12-16T20:45:34.152 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCezOur/AmELSSiiE6a38SQZw7Jntk9AxQ+ZI6ZlR1kzYJkaGTeHs35BiHZ53juUDGE1L1h9DloM08/zXDCKjtahqRMoKdmSJXfEGYsqKSGBzI9xxXSIQQfSPypd61ZjGXZsvtdTqZOh5r/yMQl7EmiCnnLbgDv5HIERhvn4cMo2ojp/nlaozPyManYEJHmbeUsJuwlArdEHa1eSGTD4CHffllirz13JKTS/nJJX4MHdV9uF377fLKmn3rf5A2dmkSp8onFTGzFxTxahnWS9Z1AEzTjcPRpz9+hJAjEbUjP4Q/JdEPOJKQvOyg+Oj+qR18E1L3lZQ28e4GT/Df4sUngWkHAzkjcJmY8VeGoTWfq/sV1SbGawxueIDEftsvjRxzc3oZDl1Qf1QpOSTMRHdothp3JuG9y7s2Kn5XA+u6G8uKRE1g0/8KaGMjQvQH0KU2mPe8Wxu/YxTuD0oVPJr22GSby8bPhYijl43LkvsqJdvn9XE3xccOQ2BMyjQIS/Uc= ceph-cf043a8a-9c53-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-16T20:45:34.240 INFO:teuthology.orchestra.run.smithi052.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCezOur/AmELSSiiE6a38SQZw7Jntk9AxQ+ZI6ZlR1kzYJkaGTeHs35BiHZ53juUDGE1L1h9DloM08/zXDCKjtahqRMoKdmSJXfEGYsqKSGBzI9xxXSIQQfSPypd61ZjGXZsvtdTqZOh5r/yMQl7EmiCnnLbgDv5HIERhvn4cMo2ojp/nlaozPyManYEJHmbeUsJuwlArdEHa1eSGTD4CHffllirz13JKTS/nJJX4MHdV9uF377fLKmn3rf5A2dmkSp8onFTGzFxTxahnWS9Z1AEzTjcPRpz9+hJAjEbUjP4Q/JdEPOJKQvOyg+Oj+qR18E1L3lZQ28e4GT/Df4sUngWkHAzkjcJmY8VeGoTWfq/sV1SbGawxueIDEftsvjRxzc3oZDl1Qf1QpOSTMRHdothp3JuG9y7s2Kn5XA+u6G8uKRE1g0/8KaGMjQvQH0KU2mPe8Wxu/YxTuD0oVPJr22GSby8bPhYijl43LkvsqJdvn9XE3xccOQ2BMyjQIS/Uc= ceph-cf043a8a-9c53-11ee-95a5-87774f69a715 2023-12-16T20:45:34.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCezOur/AmELSSiiE6a38SQZw7Jntk9AxQ+ZI6ZlR1kzYJkaGTeHs35BiHZ53juUDGE1L1h9DloM08/zXDCKjtahqRMoKdmSJXfEGYsqKSGBzI9xxXSIQQfSPypd61ZjGXZsvtdTqZOh5r/yMQl7EmiCnnLbgDv5HIERhvn4cMo2ojp/nlaozPyManYEJHmbeUsJuwlArdEHa1eSGTD4CHffllirz13JKTS/nJJX4MHdV9uF377fLKmn3rf5A2dmkSp8onFTGzFxTxahnWS9Z1AEzTjcPRpz9+hJAjEbUjP4Q/JdEPOJKQvOyg+Oj+qR18E1L3lZQ28e4GT/Df4sUngWkHAzkjcJmY8VeGoTWfq/sV1SbGawxueIDEftsvjRxzc3oZDl1Qf1QpOSTMRHdothp3JuG9y7s2Kn5XA+u6G8uKRE1g0/8KaGMjQvQH0KU2mPe8Wxu/YxTuD0oVPJr22GSby8bPhYijl43LkvsqJdvn9XE3xccOQ2BMyjQIS/Uc= ceph-cf043a8a-9c53-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-16T20:45:34.305 INFO:teuthology.orchestra.run.smithi150.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCezOur/AmELSSiiE6a38SQZw7Jntk9AxQ+ZI6ZlR1kzYJkaGTeHs35BiHZ53juUDGE1L1h9DloM08/zXDCKjtahqRMoKdmSJXfEGYsqKSGBzI9xxXSIQQfSPypd61ZjGXZsvtdTqZOh5r/yMQl7EmiCnnLbgDv5HIERhvn4cMo2ojp/nlaozPyManYEJHmbeUsJuwlArdEHa1eSGTD4CHffllirz13JKTS/nJJX4MHdV9uF377fLKmn3rf5A2dmkSp8onFTGzFxTxahnWS9Z1AEzTjcPRpz9+hJAjEbUjP4Q/JdEPOJKQvOyg+Oj+qR18E1L3lZQ28e4GT/Df4sUngWkHAzkjcJmY8VeGoTWfq/sV1SbGawxueIDEftsvjRxzc3oZDl1Qf1QpOSTMRHdothp3JuG9y7s2Kn5XA+u6G8uKRE1g0/8KaGMjQvQH0KU2mPe8Wxu/YxTuD0oVPJr22GSby8bPhYijl43LkvsqJdvn9XE3xccOQ2BMyjQIS/Uc= ceph-cf043a8a-9c53-11ee-95a5-87774f69a715 2023-12-16T20:45:34.320 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-12-16T20:45:34.609 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:45:34.858 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:34 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/1846516168' entity='client.admin' 2023-12-16T20:45:35.540 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.537+0000 7fc8330bf700 1 -- 172.21.15.52:0/859722376 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc82c0fb780 msgr2=0x7fc82c0fbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:35.540 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.537+0000 7fc8330bf700 1 --2- 172.21.15.52:0/859722376 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc82c0fb780 0x7fc82c0fbb90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc8200048c0 tx=0x7fc820009e30 comp rx=0 tx=0).stop 2023-12-16T20:45:35.540 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.537+0000 7fc8330bf700 1 -- 172.21.15.52:0/859722376 shutdown_connections 2023-12-16T20:45:35.540 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.537+0000 7fc8330bf700 1 --2- 172.21.15.52:0/859722376 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc82c0fb780 0x7fc82c0fbb90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:35.540 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.537+0000 7fc8330bf700 1 -- 172.21.15.52:0/859722376 >> 172.21.15.52:0/859722376 conn(0x7fc82c0f6e40 msgr2=0x7fc82c0f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:35.541 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.537+0000 7fc8330bf700 1 -- 172.21.15.52:0/859722376 shutdown_connections 2023-12-16T20:45:35.541 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.537+0000 7fc8330bf700 1 -- 172.21.15.52:0/859722376 wait complete. 2023-12-16T20:45:35.541 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.538+0000 7fc8330bf700 1 Processor -- start 2023-12-16T20:45:35.541 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.538+0000 7fc8330bf700 1 -- start start 2023-12-16T20:45:35.541 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.538+0000 7fc8330bf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc82c0fb780 0x7fc82c10f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:35.541 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.538+0000 7fc8330bf700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c065a30 con 0x7fc82c0fb780 2023-12-16T20:45:35.541 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.538+0000 7fc830e5b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc82c0fb780 0x7fc82c10f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:35.541 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.538+0000 7fc830e5b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc82c0fb780 0x7fc82c10f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47536/0 (socket says 172.21.15.52:47536) 2023-12-16T20:45:35.541 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.538+0000 7fc830e5b700 1 -- 172.21.15.52:0/868762065 learned_addr learned my addr 172.21.15.52:0/868762065 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:35.541 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.538+0000 7fc830e5b700 1 -- 172.21.15.52:0/868762065 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc820005040 con 0x7fc82c0fb780 2023-12-16T20:45:35.541 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.539+0000 7fc830e5b700 1 --2- 172.21.15.52:0/868762065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc82c0fb780 0x7fc82c10f7a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc82000b690 tx=0x7fc820009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:35.541 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.539+0000 7fc8297fa700 1 -- 172.21.15.52:0/868762065 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc820009650 con 0x7fc82c0fb780 2023-12-16T20:45:35.542 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.539+0000 7fc8297fa700 1 -- 172.21.15.52:0/868762065 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8200097b0 con 0x7fc82c0fb780 2023-12-16T20:45:35.542 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.539+0000 7fc8330bf700 1 -- 172.21.15.52:0/868762065 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc82c065c30 con 0x7fc82c0fb780 2023-12-16T20:45:35.542 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.539+0000 7fc8297fa700 1 -- 172.21.15.52:0/868762065 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc82001c5d0 con 0x7fc82c0fb780 2023-12-16T20:45:35.542 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.539+0000 7fc8330bf700 1 -- 172.21.15.52:0/868762065 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82c10ff20 con 0x7fc82c0fb780 2023-12-16T20:45:35.542 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.540+0000 7fc8297fa700 1 -- 172.21.15.52:0/868762065 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7fc820013420 con 0x7fc82c0fb780 2023-12-16T20:45:35.543 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.540+0000 7fc8297fa700 1 --2- 172.21.15.52:0/868762065 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fc81c036630 0x7fc81c038ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:35.543 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.540+0000 7fc8297fa700 1 -- 172.21.15.52:0/868762065 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc820049870 con 0x7fc82c0fb780 2023-12-16T20:45:35.543 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.540+0000 7fc8330bf700 1 -- 172.21.15.52:0/868762065 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc82c105410 con 0x7fc82c0fb780 2023-12-16T20:45:35.543 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.540+0000 7fc82bfff700 1 --2- 172.21.15.52:0/868762065 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fc81c036630 0x7fc81c038ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:35.543 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.540+0000 7fc82bfff700 1 --2- 172.21.15.52:0/868762065 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fc81c036630 0x7fc81c038ae0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc818009990 tx=0x7fc818006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:35.546 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.545+0000 7fc8297fa700 1 -- 172.21.15.52:0/868762065 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc820025400 con 0x7fc82c0fb780 2023-12-16T20:45:35.598 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.597+0000 7fc8297fa700 1 -- 172.21.15.52:0/868762065 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fc8200139d0 con 0x7fc82c0fb780 2023-12-16T20:45:35.748 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.747+0000 7fc8330bf700 1 -- 172.21.15.52:0/868762065 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fc82c04bb80 con 0x7fc82c0fb780 2023-12-16T20:45:35.751 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.750+0000 7fc8297fa700 1 -- 172.21.15.52:0/868762065 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fc820011900 con 0x7fc82c0fb780 2023-12-16T20:45:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.753+0000 7fc8330bf700 1 -- 172.21.15.52:0/868762065 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fc81c036630 msgr2=0x7fc81c038ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.753+0000 7fc8330bf700 1 --2- 172.21.15.52:0/868762065 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fc81c036630 0x7fc81c038ae0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc818009990 tx=0x7fc818006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.753+0000 7fc8330bf700 1 -- 172.21.15.52:0/868762065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc82c0fb780 msgr2=0x7fc82c10f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.753+0000 7fc8330bf700 1 --2- 172.21.15.52:0/868762065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc82c0fb780 0x7fc82c10f7a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc82000b690 tx=0x7fc820009e30 comp rx=0 tx=0).stop 2023-12-16T20:45:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.753+0000 7fc8330bf700 1 -- 172.21.15.52:0/868762065 shutdown_connections 2023-12-16T20:45:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.753+0000 7fc8330bf700 1 --2- 172.21.15.52:0/868762065 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fc81c036630 0x7fc81c038ae0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.753+0000 7fc8330bf700 1 --2- 172.21.15.52:0/868762065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc82c0fb780 0x7fc82c10f7a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.753+0000 7fc8330bf700 1 -- 172.21.15.52:0/868762065 >> 172.21.15.52:0/868762065 conn(0x7fc82c0f6e40 msgr2=0x7fc82c0f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:35.755 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.753+0000 7fc8330bf700 1 -- 172.21.15.52:0/868762065 shutdown_connections 2023-12-16T20:45:35.755 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:35.753+0000 7fc8330bf700 1 -- 172.21.15.52:0/868762065 wait complete. 2023-12-16T20:45:36.002 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-12-16T20:45:36.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-12-16T20:45:36.276 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:45:36.917 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:36 smithi052 ceph-mon[80761]: mgrmap e13: smithi052.nmkcbs(active, since 7s) 2023-12-16T20:45:36.917 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:36 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/868762065' entity='client.admin' 2023-12-16T20:45:36.917 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:36 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:36.917 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:36 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:36.917 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:36 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:45:36.917 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:36 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:36.917 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:36 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi052", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-16T20:45:36.917 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:36 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi052", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-16T20:45:36.917 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:36 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:45:37.250 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.248+0000 7f478d396700 1 -- 172.21.15.52:0/2514266489 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f47880fd3c0 msgr2=0x7f47880ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:37.251 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.248+0000 7f478d396700 1 --2- 172.21.15.52:0/2514266489 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f47880fd3c0 0x7f47880ff7a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4780009c60 tx=0x7f4780004d10 comp rx=0 tx=0).stop 2023-12-16T20:45:37.251 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.248+0000 7f478d396700 1 -- 172.21.15.52:0/2514266489 shutdown_connections 2023-12-16T20:45:37.251 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.248+0000 7f478d396700 1 --2- 172.21.15.52:0/2514266489 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f47880fd3c0 0x7f47880ff7a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:37.251 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.248+0000 7f478d396700 1 -- 172.21.15.52:0/2514266489 >> 172.21.15.52:0/2514266489 conn(0x7f47880f6e40 msgr2=0x7f47880f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:37.251 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.248+0000 7f478d396700 1 -- 172.21.15.52:0/2514266489 shutdown_connections 2023-12-16T20:45:37.251 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.248+0000 7f478d396700 1 -- 172.21.15.52:0/2514266489 wait complete. 2023-12-16T20:45:37.251 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.248+0000 7f478d396700 1 Processor -- start 2023-12-16T20:45:37.251 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.249+0000 7f478d396700 1 -- start start 2023-12-16T20:45:37.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.249+0000 7f478d396700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f47880fd3c0 0x7f478810cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:37.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.249+0000 7f478d396700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478810d010 con 0x7f47880fd3c0 2023-12-16T20:45:37.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.249+0000 7f4786ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f47880fd3c0 0x7f478810cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:37.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.249+0000 7f4786ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f47880fd3c0 0x7f478810cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47556/0 (socket says 172.21.15.52:47556) 2023-12-16T20:45:37.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.249+0000 7f4786ffd700 1 -- 172.21.15.52:0/2610873490 learned_addr learned my addr 172.21.15.52:0/2610873490 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:37.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.249+0000 7f4786ffd700 1 -- 172.21.15.52:0/2610873490 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4780009940 con 0x7f47880fd3c0 2023-12-16T20:45:37.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.249+0000 7f4786ffd700 1 --2- 172.21.15.52:0/2610873490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f47880fd3c0 0x7f478810cad0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4780015040 tx=0x7f4780004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:37.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.249+0000 7f477f7fe700 1 -- 172.21.15.52:0/2610873490 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f478001d440 con 0x7f47880fd3c0 2023-12-16T20:45:37.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.249+0000 7f477f7fe700 1 -- 172.21.15.52:0/2610873490 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f478001da50 con 0x7f47880fd3c0 2023-12-16T20:45:37.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.249+0000 7f478d396700 1 -- 172.21.15.52:0/2610873490 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478810d210 con 0x7f47880fd3c0 2023-12-16T20:45:37.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.250+0000 7f477f7fe700 1 -- 172.21.15.52:0/2610873490 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47800128e0 con 0x7f47880fd3c0 2023-12-16T20:45:37.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.250+0000 7f478d396700 1 -- 172.21.15.52:0/2610873490 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478810deb0 con 0x7f47880fd3c0 2023-12-16T20:45:37.253 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.250+0000 7f477f7fe700 1 -- 172.21.15.52:0/2610873490 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f4780019040 con 0x7f47880fd3c0 2023-12-16T20:45:37.253 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.251+0000 7f477f7fe700 1 --2- 172.21.15.52:0/2610873490 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f47740366d0 0x7f4774038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:37.253 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.251+0000 7f478d396700 1 -- 172.21.15.52:0/2610873490 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478805f270 con 0x7f47880fd3c0 2023-12-16T20:45:37.253 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.251+0000 7f477f7fe700 1 -- 172.21.15.52:0/2610873490 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f478004a120 con 0x7f47880fd3c0 2023-12-16T20:45:37.253 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.251+0000 7f47867fc700 1 --2- 172.21.15.52:0/2610873490 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f47740366d0 0x7f4774038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:37.254 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.251+0000 7f47867fc700 1 --2- 172.21.15.52:0/2610873490 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f47740366d0 0x7f4774038b80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4770009990 tx=0x7f4770006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:37.256 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.255+0000 7f477f7fe700 1 -- 172.21.15.52:0/2610873490 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4780021070 con 0x7f47880fd3c0 2023-12-16T20:45:37.464 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.463+0000 7f478d396700 1 -- 172.21.15.52:0/2610873490 --> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f478810e9f0 con 0x7f47740366d0 2023-12-16T20:45:37.467 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.466+0000 7f477f7fe700 1 -- 172.21.15.52:0/2610873490 <== mgr.14162 v2:172.21.15.52:6800/2799692744 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f478810e9f0 con 0x7f47740366d0 2023-12-16T20:45:37.469 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.468+0000 7f478d396700 1 -- 172.21.15.52:0/2610873490 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f47740366d0 msgr2=0x7f4774038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:37.469 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.468+0000 7f478d396700 1 --2- 172.21.15.52:0/2610873490 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f47740366d0 0x7f4774038b80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4770009990 tx=0x7f4770006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:37.469 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.468+0000 7f478d396700 1 -- 172.21.15.52:0/2610873490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f47880fd3c0 msgr2=0x7f478810cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:37.470 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.468+0000 7f478d396700 1 --2- 172.21.15.52:0/2610873490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f47880fd3c0 0x7f478810cad0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4780015040 tx=0x7f4780004d10 comp rx=0 tx=0).stop 2023-12-16T20:45:37.470 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.468+0000 7f478d396700 1 -- 172.21.15.52:0/2610873490 shutdown_connections 2023-12-16T20:45:37.470 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.468+0000 7f478d396700 1 --2- 172.21.15.52:0/2610873490 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f47740366d0 0x7f4774038b80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:37.470 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.468+0000 7f478d396700 1 --2- 172.21.15.52:0/2610873490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f47880fd3c0 0x7f478810cad0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:37.470 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.468+0000 7f478d396700 1 -- 172.21.15.52:0/2610873490 >> 172.21.15.52:0/2610873490 conn(0x7f47880f6e40 msgr2=0x7f47880f7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:37.470 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.468+0000 7f478d396700 1 -- 172.21.15.52:0/2610873490 shutdown_connections 2023-12-16T20:45:37.470 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:37.468+0000 7f478d396700 1 -- 172.21.15.52:0/2610873490 wait complete. 2023-12-16T20:45:37.724 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:37 smithi052 ceph-mon[80761]: Deploying daemon crash.smithi052 on smithi052 2023-12-16T20:45:37.725 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:37 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:38.058 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi150 2023-12-16T20:45:38.058 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-16T20:45:38.058 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.conf 2023-12-16T20:45:38.076 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-16T20:45:38.076 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-16T20:45:38.135 INFO:tasks.cephadm:Adding host smithi150 to orchestrator... 2023-12-16T20:45:38.135 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph orch host add smithi150 2023-12-16T20:45:38.464 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:45:38.891 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:38 smithi052 ceph-mon[80761]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:38.892 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:38 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:38.892 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:38 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:38.892 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:38 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:38.892 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:38 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:39.230 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.228+0000 7fcb71d6e700 1 -- 172.21.15.52:0/2049929497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb6c0fc880 msgr2=0x7fcb6c0fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:39.231 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.228+0000 7fcb71d6e700 1 --2- 172.21.15.52:0/2049929497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb6c0fc880 0x7fcb6c0fcc90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0048c0 tx=0x7fcb5c009e30 comp rx=0 tx=0).stop 2023-12-16T20:45:39.231 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.228+0000 7fcb71d6e700 1 -- 172.21.15.52:0/2049929497 shutdown_connections 2023-12-16T20:45:39.231 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.228+0000 7fcb71d6e700 1 --2- 172.21.15.52:0/2049929497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb6c0fc880 0x7fcb6c0fcc90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:39.231 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.228+0000 7fcb71d6e700 1 -- 172.21.15.52:0/2049929497 >> 172.21.15.52:0/2049929497 conn(0x7fcb6c0f7f40 msgr2=0x7fcb6c0fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:39.231 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.228+0000 7fcb71d6e700 1 -- 172.21.15.52:0/2049929497 shutdown_connections 2023-12-16T20:45:39.231 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.228+0000 7fcb71d6e700 1 -- 172.21.15.52:0/2049929497 wait complete. 2023-12-16T20:45:39.231 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.229+0000 7fcb71d6e700 1 Processor -- start 2023-12-16T20:45:39.231 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.229+0000 7fcb71d6e700 1 -- start start 2023-12-16T20:45:39.231 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.229+0000 7fcb71d6e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb6c0fc880 0x7fcb6c071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:39.231 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.229+0000 7fcb71d6e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c06ff90 con 0x7fcb6c0fc880 2023-12-16T20:45:39.231 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.230+0000 7fcb6b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb6c0fc880 0x7fcb6c071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:39.231 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.230+0000 7fcb6b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb6c0fc880 0x7fcb6c071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47568/0 (socket says 172.21.15.52:47568) 2023-12-16T20:45:39.231 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.230+0000 7fcb6b7fe700 1 -- 172.21.15.52:0/1650448873 learned_addr learned my addr 172.21.15.52:0/1650448873 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:39.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.230+0000 7fcb6b7fe700 1 -- 172.21.15.52:0/1650448873 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c005040 con 0x7fcb6c0fc880 2023-12-16T20:45:39.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.230+0000 7fcb6b7fe700 1 --2- 172.21.15.52:0/1650448873 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb6c0fc880 0x7fcb6c071b10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009330 tx=0x7fcb5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:39.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.230+0000 7fcb53fff700 1 -- 172.21.15.52:0/1650448873 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb5c0096b0 con 0x7fcb6c0fc880 2023-12-16T20:45:39.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.230+0000 7fcb53fff700 1 -- 172.21.15.52:0/1650448873 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb5c009810 con 0x7fcb6c0fc880 2023-12-16T20:45:39.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.230+0000 7fcb71d6e700 1 -- 172.21.15.52:0/1650448873 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6c070190 con 0x7fcb6c0fc880 2023-12-16T20:45:39.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.230+0000 7fcb53fff700 1 -- 172.21.15.52:0/1650448873 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb5c01c5b0 con 0x7fcb6c0fc880 2023-12-16T20:45:39.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.230+0000 7fcb71d6e700 1 -- 172.21.15.52:0/1650448873 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6c070680 con 0x7fcb6c0fc880 2023-12-16T20:45:39.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.231+0000 7fcb53fff700 1 -- 172.21.15.52:0/1650448873 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fcb5c01c870 con 0x7fcb6c0fc880 2023-12-16T20:45:39.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.232+0000 7fcb53fff700 1 --2- 172.21.15.52:0/1650448873 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fcb540366d0 0x7fcb54038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:39.234 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.232+0000 7fcb53fff700 1 -- 172.21.15.52:0/1650448873 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcb5c049a20 con 0x7fcb6c0fc880 2023-12-16T20:45:39.234 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.232+0000 7fcb6affd700 1 --2- 172.21.15.52:0/1650448873 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fcb540366d0 0x7fcb54038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:39.234 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.232+0000 7fcb71d6e700 1 -- 172.21.15.52:0/1650448873 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6c106510 con 0x7fcb6c0fc880 2023-12-16T20:45:39.234 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.232+0000 7fcb6affd700 1 --2- 172.21.15.52:0/1650448873 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fcb540366d0 0x7fcb54038b80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcb60009990 tx=0x7fcb60006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:39.238 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.237+0000 7fcb53fff700 1 -- 172.21.15.52:0/1650448873 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fcb5c011dd0 con 0x7fcb6c0fc880 2023-12-16T20:45:39.452 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:39.450+0000 7fcb71d6e700 1 -- 172.21.15.52:0/1650448873 --> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi150", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb6c0708c0 con 0x7fcb540366d0 2023-12-16T20:45:39.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:39 smithi052 ceph-mon[80761]: Deploying daemon node-exporter.smithi052 on smithi052 2023-12-16T20:45:39.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:39 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:41.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:40 smithi052 ceph-mon[80761]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi150", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:41.441 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:41.439+0000 7fcb53fff700 1 -- 172.21.15.52:0/1650448873 <== mgr.14162 v2:172.21.15.52:6800/2799692744 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fcb6c0708c0 con 0x7fcb540366d0 2023-12-16T20:45:41.441 INFO:teuthology.orchestra.run.smithi052.stdout:Added host 'smithi150' with addr '172.21.15.150' 2023-12-16T20:45:41.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:41.441+0000 7fcb71d6e700 1 -- 172.21.15.52:0/1650448873 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fcb540366d0 msgr2=0x7fcb54038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:41.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:41.441+0000 7fcb71d6e700 1 --2- 172.21.15.52:0/1650448873 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fcb540366d0 0x7fcb54038b80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcb60009990 tx=0x7fcb60006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:41.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:41.441+0000 7fcb71d6e700 1 -- 172.21.15.52:0/1650448873 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb6c0fc880 msgr2=0x7fcb6c071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:41.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:41.441+0000 7fcb71d6e700 1 --2- 172.21.15.52:0/1650448873 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb6c0fc880 0x7fcb6c071b10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009330 tx=0x7fcb5c009e30 comp rx=0 tx=0).stop 2023-12-16T20:45:41.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:41.441+0000 7fcb71d6e700 1 -- 172.21.15.52:0/1650448873 shutdown_connections 2023-12-16T20:45:41.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:41.441+0000 7fcb71d6e700 1 --2- 172.21.15.52:0/1650448873 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fcb540366d0 0x7fcb54038b80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:41.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:41.441+0000 7fcb71d6e700 1 --2- 172.21.15.52:0/1650448873 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb6c0fc880 0x7fcb6c071b10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:41.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:41.441+0000 7fcb71d6e700 1 -- 172.21.15.52:0/1650448873 >> 172.21.15.52:0/1650448873 conn(0x7fcb6c0f7f40 msgr2=0x7fcb6c0f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:41.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:41.442+0000 7fcb71d6e700 1 -- 172.21.15.52:0/1650448873 shutdown_connections 2023-12-16T20:45:41.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:41.442+0000 7fcb71d6e700 1 -- 172.21.15.52:0/1650448873 wait complete. 2023-12-16T20:45:41.692 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:41 smithi052 ceph-mon[80761]: Deploying cephadm binary to smithi150 2023-12-16T20:45:41.692 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:41 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:41.986 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph orch host ls --format=json 2023-12-16T20:45:42.463 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:45:42.957 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:42 smithi052 ceph-mon[80761]: Added host smithi150 2023-12-16T20:45:42.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:42 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:42.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:42 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:42.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:42 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:42.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:42 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:43.310 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.309+0000 7f9df55f3700 1 -- 172.21.15.52:0/647823345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9df00fb790 msgr2=0x7f9df00fbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:43.310 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.309+0000 7f9df55f3700 1 --2- 172.21.15.52:0/647823345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9df00fb790 0x7f9df00fbba0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9de80048c0 tx=0x7f9de8009e30 comp rx=0 tx=0).stop 2023-12-16T20:45:43.310 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.309+0000 7f9df55f3700 1 -- 172.21.15.52:0/647823345 shutdown_connections 2023-12-16T20:45:43.311 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.309+0000 7f9df55f3700 1 --2- 172.21.15.52:0/647823345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9df00fb790 0x7f9df00fbba0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:43.311 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.309+0000 7f9df55f3700 1 -- 172.21.15.52:0/647823345 >> 172.21.15.52:0/647823345 conn(0x7f9df00f6e50 msgr2=0x7f9df00f9280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:43.311 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.309+0000 7f9df55f3700 1 -- 172.21.15.52:0/647823345 shutdown_connections 2023-12-16T20:45:43.311 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.309+0000 7f9df55f3700 1 -- 172.21.15.52:0/647823345 wait complete. 2023-12-16T20:45:43.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.310+0000 7f9df55f3700 1 Processor -- start 2023-12-16T20:45:43.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.310+0000 7f9df55f3700 1 -- start start 2023-12-16T20:45:43.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.310+0000 7f9df55f3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9df00fb790 0x7f9df010cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:43.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.310+0000 7f9df55f3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df010d020 con 0x7f9df00fb790 2023-12-16T20:45:43.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.310+0000 7f9deeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9df00fb790 0x7f9df010cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:43.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.310+0000 7f9deeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9df00fb790 0x7f9df010cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58288/0 (socket says 172.21.15.52:58288) 2023-12-16T20:45:43.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.310+0000 7f9deeffd700 1 -- 172.21.15.52:0/1114733496 learned_addr learned my addr 172.21.15.52:0/1114733496 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:43.313 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.310+0000 7f9deeffd700 1 -- 172.21.15.52:0/1114733496 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de8005040 con 0x7f9df00fb790 2023-12-16T20:45:43.313 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.311+0000 7f9deeffd700 1 --2- 172.21.15.52:0/1114733496 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9df00fb790 0x7f9df010cae0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009330 tx=0x7f9de8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:43.313 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.311+0000 7f9de77fe700 1 -- 172.21.15.52:0/1114733496 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9de80096b0 con 0x7f9df00fb790 2023-12-16T20:45:43.313 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.311+0000 7f9de77fe700 1 -- 172.21.15.52:0/1114733496 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9de8009810 con 0x7f9df00fb790 2023-12-16T20:45:43.313 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.311+0000 7f9df55f3700 1 -- 172.21.15.52:0/1114733496 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df010d220 con 0x7f9df00fb790 2023-12-16T20:45:43.313 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.311+0000 7f9de77fe700 1 -- 172.21.15.52:0/1114733496 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9de801c5b0 con 0x7f9df00fb790 2023-12-16T20:45:43.313 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.311+0000 7f9df55f3700 1 -- 172.21.15.52:0/1114733496 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df010dec0 con 0x7f9df00fb790 2023-12-16T20:45:43.314 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.312+0000 7f9de77fe700 1 -- 172.21.15.52:0/1114733496 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f9de8013420 con 0x7f9df00fb790 2023-12-16T20:45:43.314 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.312+0000 7f9df55f3700 1 -- 172.21.15.52:0/1114733496 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df005f270 con 0x7f9df00fb790 2023-12-16T20:45:43.314 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.312+0000 7f9de77fe700 1 --2- 172.21.15.52:0/1114733496 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f9ddc036720 0x7f9ddc038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:43.314 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.312+0000 7f9de77fe700 1 -- 172.21.15.52:0/1114733496 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9de80499b0 con 0x7f9df00fb790 2023-12-16T20:45:43.314 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.312+0000 7f9dee7fc700 1 --2- 172.21.15.52:0/1114733496 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f9ddc036720 0x7f9ddc038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:43.314 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.312+0000 7f9dee7fc700 1 --2- 172.21.15.52:0/1114733496 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f9ddc036720 0x7f9ddc038bd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8009990 tx=0x7f9dd8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:43.318 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.317+0000 7f9de77fe700 1 -- 172.21.15.52:0/1114733496 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9de80119c0 con 0x7f9df00fb790 2023-12-16T20:45:43.525 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.524+0000 7f9df55f3700 1 -- 172.21.15.52:0/1114733496 --> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9df010ea80 con 0x7f9ddc036720 2023-12-16T20:45:43.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.525+0000 7f9de77fe700 1 -- 172.21.15.52:0/1114733496 <== mgr.14162 v2:172.21.15.52:6800/2799692744 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f9df010ea80 con 0x7f9ddc036720 2023-12-16T20:45:43.526 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:45:43.526 INFO:teuthology.orchestra.run.smithi052.stdout:[{"addr": "172.21.15.52", "hostname": "smithi052", "labels": [], "status": ""}, {"addr": "172.21.15.150", "hostname": "smithi150", "labels": [], "status": ""}] 2023-12-16T20:45:43.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.527+0000 7f9df55f3700 1 -- 172.21.15.52:0/1114733496 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f9ddc036720 msgr2=0x7f9ddc038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:43.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.527+0000 7f9df55f3700 1 --2- 172.21.15.52:0/1114733496 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f9ddc036720 0x7f9ddc038bd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8009990 tx=0x7f9dd8006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:43.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.527+0000 7f9df55f3700 1 -- 172.21.15.52:0/1114733496 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9df00fb790 msgr2=0x7f9df010cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:43.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.527+0000 7f9df55f3700 1 --2- 172.21.15.52:0/1114733496 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9df00fb790 0x7f9df010cae0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009330 tx=0x7f9de8009e30 comp rx=0 tx=0).stop 2023-12-16T20:45:43.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.527+0000 7f9df55f3700 1 -- 172.21.15.52:0/1114733496 shutdown_connections 2023-12-16T20:45:43.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.527+0000 7f9df55f3700 1 --2- 172.21.15.52:0/1114733496 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f9ddc036720 0x7f9ddc038bd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:43.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.527+0000 7f9df55f3700 1 --2- 172.21.15.52:0/1114733496 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9df00fb790 0x7f9df010cae0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:43.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.527+0000 7f9df55f3700 1 -- 172.21.15.52:0/1114733496 >> 172.21.15.52:0/1114733496 conn(0x7f9df00f6e50 msgr2=0x7f9df0066120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:43.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.527+0000 7f9df55f3700 1 -- 172.21.15.52:0/1114733496 shutdown_connections 2023-12-16T20:45:43.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:43.527+0000 7f9df55f3700 1 -- 172.21.15.52:0/1114733496 wait complete. 2023-12-16T20:45:43.934 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:43 smithi052 ceph-mon[80761]: Deploying daemon alertmanager.smithi052 on smithi052 2023-12-16T20:45:43.958 INFO:tasks.cephadm:Setting crush tunables to default 2023-12-16T20:45:43.958 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph osd crush tunables default 2023-12-16T20:45:44.220 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:45:45.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:44 smithi052 ceph-mon[80761]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T20:45:45.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:44 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:45.621 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.619+0000 7f99fbfc3700 1 -- 172.21.15.52:0/408171452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99f4065aa0 msgr2=0x7f99f4101830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:45.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.619+0000 7f99fbfc3700 1 --2- 172.21.15.52:0/408171452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99f4065aa0 0x7f99f4101830 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f99f00048c0 tx=0x7f99f0009e30 comp rx=0 tx=0).stop 2023-12-16T20:45:45.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.619+0000 7f99fbfc3700 1 -- 172.21.15.52:0/408171452 shutdown_connections 2023-12-16T20:45:45.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.619+0000 7f99fbfc3700 1 --2- 172.21.15.52:0/408171452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99f4065aa0 0x7f99f4101830 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:45.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.619+0000 7f99fbfc3700 1 -- 172.21.15.52:0/408171452 >> 172.21.15.52:0/408171452 conn(0x7f99f40f6e40 msgr2=0x7f99f40f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:45.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.619+0000 7f99fbfc3700 1 -- 172.21.15.52:0/408171452 shutdown_connections 2023-12-16T20:45:45.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.619+0000 7f99fbfc3700 1 -- 172.21.15.52:0/408171452 wait complete. 2023-12-16T20:45:45.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.620+0000 7f99fbfc3700 1 Processor -- start 2023-12-16T20:45:45.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.620+0000 7f99fbfc3700 1 -- start start 2023-12-16T20:45:45.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.620+0000 7f99fbfc3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99f4065aa0 0x7f99f410ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:45.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.620+0000 7f99fbfc3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f410f230 con 0x7f99f4065aa0 2023-12-16T20:45:45.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.620+0000 7f99f9d5f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99f4065aa0 0x7f99f410ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:45.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.620+0000 7f99f9d5f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99f4065aa0 0x7f99f410ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58298/0 (socket says 172.21.15.52:58298) 2023-12-16T20:45:45.623 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.620+0000 7f99f9d5f700 1 -- 172.21.15.52:0/4221185356 learned_addr learned my addr 172.21.15.52:0/4221185356 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:45:45.623 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.620+0000 7f99f9d5f700 1 -- 172.21.15.52:0/4221185356 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f0005040 con 0x7f99f4065aa0 2023-12-16T20:45:45.623 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.621+0000 7f99f9d5f700 1 --2- 172.21.15.52:0/4221185356 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99f4065aa0 0x7f99f410ecf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009040 tx=0x7f99f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:45.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.621+0000 7f99ea7fc700 1 -- 172.21.15.52:0/4221185356 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99f0009620 con 0x7f99f4065aa0 2023-12-16T20:45:45.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.621+0000 7f99ea7fc700 1 -- 172.21.15.52:0/4221185356 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99f0009780 con 0x7f99f4065aa0 2023-12-16T20:45:45.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.621+0000 7f99fbfc3700 1 -- 172.21.15.52:0/4221185356 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f410f430 con 0x7f99f4065aa0 2023-12-16T20:45:45.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.621+0000 7f99ea7fc700 1 -- 172.21.15.52:0/4221185356 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99f001c5b0 con 0x7f99f4065aa0 2023-12-16T20:45:45.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.621+0000 7f99fbfc3700 1 -- 172.21.15.52:0/4221185356 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f410ff90 con 0x7f99f4065aa0 2023-12-16T20:45:45.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.622+0000 7f99ea7fc700 1 -- 172.21.15.52:0/4221185356 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f99f001c870 con 0x7f99f4065aa0 2023-12-16T20:45:45.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.622+0000 7f99ea7fc700 1 --2- 172.21.15.52:0/4221185356 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f99e00366d0 0x7f99e0038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:45.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.622+0000 7f99ea7fc700 1 -- 172.21.15.52:0/4221185356 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f99f0049850 con 0x7f99f4065aa0 2023-12-16T20:45:45.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.622+0000 7f99fbfc3700 1 -- 172.21.15.52:0/4221185356 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f404bb80 con 0x7f99f4065aa0 2023-12-16T20:45:45.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.622+0000 7f99f955e700 1 --2- 172.21.15.52:0/4221185356 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f99e00366d0 0x7f99e0038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:45.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.623+0000 7f99f955e700 1 --2- 172.21.15.52:0/4221185356 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f99e00366d0 0x7f99e0038b80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009990 tx=0x7f99e4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:45.628 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.627+0000 7f99ea7fc700 1 -- 172.21.15.52:0/4221185356 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f99f0023300 con 0x7f99f4065aa0 2023-12-16T20:45:45.872 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.870+0000 7f99fbfc3700 1 -- 172.21.15.52:0/4221185356 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f99f41046f0 con 0x7f99f4065aa0 2023-12-16T20:45:45.925 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.924+0000 7f99ea7fc700 1 -- 172.21.15.52:0/4221185356 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f99f0011b60 con 0x7f99f4065aa0 2023-12-16T20:45:45.927 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.926+0000 7f99fbfc3700 1 -- 172.21.15.52:0/4221185356 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f99e00366d0 msgr2=0x7f99e0038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:45.928 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.926+0000 7f99fbfc3700 1 --2- 172.21.15.52:0/4221185356 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f99e00366d0 0x7f99e0038b80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009990 tx=0x7f99e4006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:45.928 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.926+0000 7f99fbfc3700 1 -- 172.21.15.52:0/4221185356 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99f4065aa0 msgr2=0x7f99f410ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:45.928 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.926+0000 7f99fbfc3700 1 --2- 172.21.15.52:0/4221185356 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99f4065aa0 0x7f99f410ecf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009040 tx=0x7f99f0009e30 comp rx=0 tx=0).stop 2023-12-16T20:45:45.928 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.926+0000 7f99fbfc3700 1 -- 172.21.15.52:0/4221185356 shutdown_connections 2023-12-16T20:45:45.928 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.926+0000 7f99fbfc3700 1 --2- 172.21.15.52:0/4221185356 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f99e00366d0 0x7f99e0038b80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:45.928 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.926+0000 7f99fbfc3700 1 --2- 172.21.15.52:0/4221185356 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99f4065aa0 0x7f99f410ecf0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:45.928 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.926+0000 7f99fbfc3700 1 -- 172.21.15.52:0/4221185356 >> 172.21.15.52:0/4221185356 conn(0x7f99f40f6e40 msgr2=0x7f99f40f7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:45.928 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.927+0000 7f99fbfc3700 1 -- 172.21.15.52:0/4221185356 shutdown_connections 2023-12-16T20:45:45.928 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:45:45.927+0000 7f99fbfc3700 1 -- 172.21.15.52:0/4221185356 wait complete. 2023-12-16T20:45:45.929 INFO:teuthology.orchestra.run.smithi052.stderr:adjusted tunables profile to default 2023-12-16T20:45:46.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:45 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/4221185356' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-16T20:45:46.847 INFO:tasks.cephadm:Adding mon.smithi052 on smithi052 2023-12-16T20:45:46.847 INFO:tasks.cephadm:Adding mon.smithi150 on smithi150 2023-12-16T20:45:46.847 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph orch apply mon '2;smithi052:172.21.15.52=smithi052;smithi150:172.21.15.150=smithi150' 2023-12-16T20:45:47.093 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:45:47.139 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:45:47.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:46 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/4221185356' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-16T20:45:47.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:46 smithi052 ceph-mon[80761]: osdmap e4: 0 total, 0 up, 0 in 2023-12-16T20:45:48.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.405+0000 7f9f874e3700 1 -- 172.21.15.150:0/2019576815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f800fb770 msgr2=0x7f9f800fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:48.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.405+0000 7f9f874e3700 1 --2- 172.21.15.150:0/2019576815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f800fb770 0x7f9f800fbb80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9f700048c0 tx=0x7f9f70009e30 comp rx=0 tx=0).stop 2023-12-16T20:45:48.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.405+0000 7f9f874e3700 1 -- 172.21.15.150:0/2019576815 shutdown_connections 2023-12-16T20:45:48.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.405+0000 7f9f874e3700 1 --2- 172.21.15.150:0/2019576815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f800fb770 0x7f9f800fbb80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:48.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.405+0000 7f9f874e3700 1 -- 172.21.15.150:0/2019576815 >> 172.21.15.150:0/2019576815 conn(0x7f9f800f6e50 msgr2=0x7f9f800f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:48.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.405+0000 7f9f874e3700 1 -- 172.21.15.150:0/2019576815 shutdown_connections 2023-12-16T20:45:48.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.405+0000 7f9f874e3700 1 -- 172.21.15.150:0/2019576815 wait complete. 2023-12-16T20:45:48.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.405+0000 7f9f874e3700 1 Processor -- start 2023-12-16T20:45:48.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.405+0000 7f9f874e3700 1 -- start start 2023-12-16T20:45:48.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.406+0000 7f9f874e3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f800fb770 0x7f9f8010b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:48.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.406+0000 7f9f874e3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8010bed0 con 0x7f9f800fb770 2023-12-16T20:45:48.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.406+0000 7f9f8527f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f800fb770 0x7f9f8010b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:48.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.406+0000 7f9f8527f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f800fb770 0x7f9f8010b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:43772/0 (socket says 172.21.15.150:43772) 2023-12-16T20:45:48.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.406+0000 7f9f8527f700 1 -- 172.21.15.150:0/3043455722 learned_addr learned my addr 172.21.15.150:0/3043455722 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:45:48.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.406+0000 7f9f8527f700 1 -- 172.21.15.150:0/3043455722 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f70005040 con 0x7f9f800fb770 2023-12-16T20:45:48.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.406+0000 7f9f8527f700 1 --2- 172.21.15.150:0/3043455722 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f800fb770 0x7f9f8010b990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9f70009680 tx=0x7f9f70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:48.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.406+0000 7f9f75ffb700 1 -- 172.21.15.150:0/3043455722 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f70004e60 con 0x7f9f800fb770 2023-12-16T20:45:48.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.406+0000 7f9f75ffb700 1 -- 172.21.15.150:0/3043455722 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f7000be50 con 0x7f9f800fb770 2023-12-16T20:45:48.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.406+0000 7f9f874e3700 1 -- 172.21.15.150:0/3043455722 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8010c0d0 con 0x7f9f800fb770 2023-12-16T20:45:48.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.406+0000 7f9f75ffb700 1 -- 172.21.15.150:0/3043455722 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f7001e6b0 con 0x7f9f800fb770 2023-12-16T20:45:48.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.407+0000 7f9f874e3700 1 -- 172.21.15.150:0/3043455722 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8010cd70 con 0x7f9f800fb770 2023-12-16T20:45:48.410 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.407+0000 7f9f75ffb700 1 -- 172.21.15.150:0/3043455722 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f9f70013630 con 0x7f9f800fb770 2023-12-16T20:45:48.410 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.407+0000 7f9f874e3700 1 -- 172.21.15.150:0/3043455722 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8005f270 con 0x7f9f800fb770 2023-12-16T20:45:48.410 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.408+0000 7f9f75ffb700 1 --2- 172.21.15.150:0/3043455722 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f9f6c0366d0 0x7f9f6c038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:48.410 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.408+0000 7f9f75ffb700 1 -- 172.21.15.150:0/3043455722 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9f7004c580 con 0x7f9f800fb770 2023-12-16T20:45:48.410 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.408+0000 7f9f84a7e700 1 --2- 172.21.15.150:0/3043455722 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f9f6c0366d0 0x7f9f6c038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:48.411 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.408+0000 7f9f84a7e700 1 --2- 172.21.15.150:0/3043455722 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f9f6c0366d0 0x7f9f6c038b80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c009990 tx=0x7f9f7c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:48.415 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.413+0000 7f9f75ffb700 1 -- 172.21.15.150:0/3043455722 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9f700119c0 con 0x7f9f800fb770 2023-12-16T20:45:48.611 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.610+0000 7f9f874e3700 1 -- 172.21.15.150:0/3043455722 --> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi052:172.21.15.52=smithi052;smithi150:172.21.15.150=smithi150", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f8010d9a0 con 0x7f9f6c0366d0 2023-12-16T20:45:48.617 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.616+0000 7f9f75ffb700 1 -- 172.21.15.150:0/3043455722 <== mgr.14162 v2:172.21.15.52:6800/2799692744 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9f8010d9a0 con 0x7f9f6c0366d0 2023-12-16T20:45:48.618 INFO:teuthology.orchestra.run.smithi150.stdout:Scheduled mon update... 2023-12-16T20:45:48.620 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.618+0000 7f9f874e3700 1 -- 172.21.15.150:0/3043455722 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f9f6c0366d0 msgr2=0x7f9f6c038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:48.620 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.618+0000 7f9f874e3700 1 --2- 172.21.15.150:0/3043455722 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f9f6c0366d0 0x7f9f6c038b80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c009990 tx=0x7f9f7c006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:48.620 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.618+0000 7f9f874e3700 1 -- 172.21.15.150:0/3043455722 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f800fb770 msgr2=0x7f9f8010b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:48.620 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.618+0000 7f9f874e3700 1 --2- 172.21.15.150:0/3043455722 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f800fb770 0x7f9f8010b990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9f70009680 tx=0x7f9f70009e30 comp rx=0 tx=0).stop 2023-12-16T20:45:48.620 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.618+0000 7f9f874e3700 1 -- 172.21.15.150:0/3043455722 shutdown_connections 2023-12-16T20:45:48.620 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.618+0000 7f9f874e3700 1 --2- 172.21.15.150:0/3043455722 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f9f6c0366d0 0x7f9f6c038b80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:48.620 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.618+0000 7f9f874e3700 1 --2- 172.21.15.150:0/3043455722 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f800fb770 0x7f9f8010b990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:48.620 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.618+0000 7f9f874e3700 1 -- 172.21.15.150:0/3043455722 >> 172.21.15.150:0/3043455722 conn(0x7f9f800f6e50 msgr2=0x7f9f800f7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:48.620 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.618+0000 7f9f874e3700 1 -- 172.21.15.150:0/3043455722 shutdown_connections 2023-12-16T20:45:48.621 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:48.618+0000 7f9f874e3700 1 -- 172.21.15.150:0/3043455722 wait complete. 2023-12-16T20:45:48.872 DEBUG:teuthology.orchestra.run.smithi150:mon.smithi150> sudo journalctl -f -n 0 -u ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@mon.smithi150.service 2023-12-16T20:45:48.874 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:45:48.874 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:45:49.227 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:45:49.273 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:45:49.926 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.923+0000 7f9465612700 1 -- 172.21.15.150:0/1357620773 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94600fb770 msgr2=0x7f94600fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:49.926 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.923+0000 7f9465612700 1 --2- 172.21.15.150:0/1357620773 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94600fb770 0x7f94600fbb80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f94500048c0 tx=0x7f9450009e30 comp rx=0 tx=0).stop 2023-12-16T20:45:49.926 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.923+0000 7f9465612700 1 -- 172.21.15.150:0/1357620773 shutdown_connections 2023-12-16T20:45:49.926 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.923+0000 7f9465612700 1 --2- 172.21.15.150:0/1357620773 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94600fb770 0x7f94600fbb80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:49.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.923+0000 7f9465612700 1 -- 172.21.15.150:0/1357620773 >> 172.21.15.150:0/1357620773 conn(0x7f94600f6e50 msgr2=0x7f94600f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:49.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.923+0000 7f9465612700 1 -- 172.21.15.150:0/1357620773 shutdown_connections 2023-12-16T20:45:49.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.923+0000 7f9465612700 1 -- 172.21.15.150:0/1357620773 wait complete. 2023-12-16T20:45:49.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.924+0000 7f9465612700 1 Processor -- start 2023-12-16T20:45:49.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.924+0000 7f9465612700 1 -- start start 2023-12-16T20:45:49.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.924+0000 7f9465612700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94600fb770 0x7f946010b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:49.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.924+0000 7f9465612700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946010bf00 con 0x7f94600fb770 2023-12-16T20:45:49.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.925+0000 7f945effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94600fb770 0x7f946010b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:49.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.925+0000 7f945effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94600fb770 0x7f946010b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:41232/0 (socket says 172.21.15.150:41232) 2023-12-16T20:45:49.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.925+0000 7f945effd700 1 -- 172.21.15.150:0/1100037125 learned_addr learned my addr 172.21.15.150:0/1100037125 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:45:49.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.925+0000 7f945effd700 1 -- 172.21.15.150:0/1100037125 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9450005040 con 0x7f94600fb770 2023-12-16T20:45:49.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.925+0000 7f945effd700 1 --2- 172.21.15.150:0/1100037125 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94600fb770 0x7f946010b9c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9450009330 tx=0x7f9450009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:49.928 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.925+0000 7f9457fff700 1 -- 172.21.15.150:0/1100037125 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94500096b0 con 0x7f94600fb770 2023-12-16T20:45:49.928 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.925+0000 7f9465612700 1 -- 172.21.15.150:0/1100037125 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946010c100 con 0x7f94600fb770 2023-12-16T20:45:49.929 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.925+0000 7f9457fff700 1 -- 172.21.15.150:0/1100037125 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9450009810 con 0x7f94600fb770 2023-12-16T20:45:49.929 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.925+0000 7f9457fff700 1 -- 172.21.15.150:0/1100037125 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f945001c5b0 con 0x7f94600fb770 2023-12-16T20:45:49.929 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.925+0000 7f9465612700 1 -- 172.21.15.150:0/1100037125 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946010cda0 con 0x7f94600fb770 2023-12-16T20:45:49.929 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.926+0000 7f9457fff700 1 -- 172.21.15.150:0/1100037125 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f9450013420 con 0x7f94600fb770 2023-12-16T20:45:49.929 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.926+0000 7f9465612700 1 -- 172.21.15.150:0/1100037125 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946005f270 con 0x7f94600fb770 2023-12-16T20:45:49.929 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.926+0000 7f9457fff700 1 --2- 172.21.15.150:0/1100037125 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f944c0366d0 0x7f944c038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:49.929 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.926+0000 7f9457fff700 1 -- 172.21.15.150:0/1100037125 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9450049ac0 con 0x7f94600fb770 2023-12-16T20:45:49.929 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.926+0000 7f94567fc700 1 --2- 172.21.15.150:0/1100037125 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f944c0366d0 0x7f944c038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:49.929 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.927+0000 7f94567fc700 1 --2- 172.21.15.150:0/1100037125 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f944c0366d0 0x7f944c038b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9448009990 tx=0x7f9448006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:49.933 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:49.931+0000 7f9457fff700 1 -- 172.21.15.150:0/1100037125 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f94500119c0 con 0x7f94600fb770 2023-12-16T20:45:50.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:49 smithi052 ceph-mon[80761]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:45:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:49 smithi052 ceph-mon[80761]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi052:172.21.15.52=smithi052;smithi150:172.21.15.150=smithi150", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:45:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:49 smithi052 ceph-mon[80761]: Saving service mon spec with placement smithi052:172.21.15.52=smithi052;smithi150:172.21.15.150=smithi150;count:2 2023-12-16T20:45:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:49 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:49 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:49 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:49 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:49 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:49 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:49 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:49 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T20:45:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:49 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:50.192 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:50.190+0000 7f9465612700 1 -- 172.21.15.150:0/1100037125 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f946010d8f0 con 0x7f94600fb770 2023-12-16T20:45:50.192 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:50.191+0000 7f9457fff700 1 -- 172.21.15.150:0/1100037125 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9450011370 con 0x7f94600fb770 2023-12-16T20:45:50.193 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:45:50.193 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:45:50.194 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:50.193+0000 7f9465612700 1 -- 172.21.15.150:0/1100037125 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f944c0366d0 msgr2=0x7f944c038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:50.194 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:50.193+0000 7f9465612700 1 --2- 172.21.15.150:0/1100037125 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f944c0366d0 0x7f944c038b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9448009990 tx=0x7f9448006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:50.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:50.193+0000 7f9465612700 1 -- 172.21.15.150:0/1100037125 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94600fb770 msgr2=0x7f946010b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:50.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:50.193+0000 7f9465612700 1 --2- 172.21.15.150:0/1100037125 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94600fb770 0x7f946010b9c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9450009330 tx=0x7f9450009e30 comp rx=0 tx=0).stop 2023-12-16T20:45:50.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:50.193+0000 7f9465612700 1 -- 172.21.15.150:0/1100037125 shutdown_connections 2023-12-16T20:45:50.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:50.193+0000 7f9465612700 1 --2- 172.21.15.150:0/1100037125 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f944c0366d0 0x7f944c038b80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:50.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:50.193+0000 7f9465612700 1 --2- 172.21.15.150:0/1100037125 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94600fb770 0x7f946010b9c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:50.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:50.193+0000 7f9465612700 1 -- 172.21.15.150:0/1100037125 >> 172.21.15.150:0/1100037125 conn(0x7f94600f6e50 msgr2=0x7f94600f7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:50.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:50.193+0000 7f9465612700 1 -- 172.21.15.150:0/1100037125 shutdown_connections 2023-12-16T20:45:50.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:50.193+0000 7f9465612700 1 -- 172.21.15.150:0/1100037125 wait complete. 2023-12-16T20:45:50.195 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:45:51.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:50 smithi052 ceph-mon[80761]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T20:45:51.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:50 smithi052 ceph-mon[80761]: Deploying daemon grafana.smithi052 on smithi052 2023-12-16T20:45:51.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:50 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/1100037125' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:45:51.437 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:45:51.437 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:45:51.683 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:45:51.729 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:45:52.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:51 smithi052 ceph-mon[80761]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:45:52.672 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.670+0000 7faba54a0700 1 -- 172.21.15.150:0/1369274259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faba00fc880 msgr2=0x7faba00fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:52.672 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.670+0000 7faba54a0700 1 --2- 172.21.15.150:0/1369274259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faba00fc880 0x7faba00fcc90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fab900048c0 tx=0x7fab90009e30 comp rx=0 tx=0).stop 2023-12-16T20:45:52.672 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.670+0000 7faba54a0700 1 -- 172.21.15.150:0/1369274259 shutdown_connections 2023-12-16T20:45:52.673 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.670+0000 7faba54a0700 1 --2- 172.21.15.150:0/1369274259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faba00fc880 0x7faba00fcc90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:52.673 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.670+0000 7faba54a0700 1 -- 172.21.15.150:0/1369274259 >> 172.21.15.150:0/1369274259 conn(0x7faba00f7f40 msgr2=0x7faba00fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:52.673 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.671+0000 7faba54a0700 1 -- 172.21.15.150:0/1369274259 shutdown_connections 2023-12-16T20:45:52.673 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.671+0000 7faba54a0700 1 -- 172.21.15.150:0/1369274259 wait complete. 2023-12-16T20:45:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.671+0000 7faba54a0700 1 Processor -- start 2023-12-16T20:45:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.671+0000 7faba54a0700 1 -- start start 2023-12-16T20:45:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.671+0000 7faba54a0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faba00fc880 0x7faba010cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.671+0000 7faba54a0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba010d000 con 0x7faba00fc880 2023-12-16T20:45:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.672+0000 7fab9effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faba00fc880 0x7faba010cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.672+0000 7fab9effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faba00fc880 0x7faba010cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:41238/0 (socket says 172.21.15.150:41238) 2023-12-16T20:45:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.672+0000 7fab9effd700 1 -- 172.21.15.150:0/2338625064 learned_addr learned my addr 172.21.15.150:0/2338625064 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:45:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.672+0000 7fab9effd700 1 -- 172.21.15.150:0/2338625064 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab90005040 con 0x7faba00fc880 2023-12-16T20:45:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.672+0000 7fab9effd700 1 --2- 172.21.15.150:0/2338625064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faba00fc880 0x7faba010cac0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fab90009330 tx=0x7fab90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.672+0000 7fab877fe700 1 -- 172.21.15.150:0/2338625064 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab900096b0 con 0x7faba00fc880 2023-12-16T20:45:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.672+0000 7fab877fe700 1 -- 172.21.15.150:0/2338625064 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab90009810 con 0x7faba00fc880 2023-12-16T20:45:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.672+0000 7faba54a0700 1 -- 172.21.15.150:0/2338625064 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba010d200 con 0x7faba00fc880 2023-12-16T20:45:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.672+0000 7fab877fe700 1 -- 172.21.15.150:0/2338625064 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab9001c5b0 con 0x7faba00fc880 2023-12-16T20:45:52.676 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.672+0000 7faba54a0700 1 -- 172.21.15.150:0/2338625064 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba010dea0 con 0x7faba00fc880 2023-12-16T20:45:52.676 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.673+0000 7fab877fe700 1 -- 172.21.15.150:0/2338625064 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fab90013420 con 0x7faba00fc880 2023-12-16T20:45:52.676 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.673+0000 7faba54a0700 1 -- 172.21.15.150:0/2338625064 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba005f270 con 0x7faba00fc880 2023-12-16T20:45:52.676 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.673+0000 7fab877fe700 1 --2- 172.21.15.150:0/2338625064 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fab880366d0 0x7fab88038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:52.676 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.673+0000 7fab877fe700 1 -- 172.21.15.150:0/2338625064 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fab90049ac0 con 0x7faba00fc880 2023-12-16T20:45:52.676 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.673+0000 7fab9e7fc700 1 --2- 172.21.15.150:0/2338625064 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fab880366d0 0x7fab88038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:52.676 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.674+0000 7fab9e7fc700 1 --2- 172.21.15.150:0/2338625064 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fab880366d0 0x7fab88038b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fab94009990 tx=0x7fab94006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:52.680 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.678+0000 7fab877fe700 1 -- 172.21.15.150:0/2338625064 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fab900119c0 con 0x7faba00fc880 2023-12-16T20:45:52.939 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.937+0000 7faba54a0700 1 -- 172.21.15.150:0/2338625064 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faba010e9f0 con 0x7faba00fc880 2023-12-16T20:45:52.939 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.938+0000 7fab877fe700 1 -- 172.21.15.150:0/2338625064 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fab90011370 con 0x7faba00fc880 2023-12-16T20:45:52.940 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:45:52.940 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:45:52.941 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.940+0000 7faba54a0700 1 -- 172.21.15.150:0/2338625064 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fab880366d0 msgr2=0x7fab88038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:52.941 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.940+0000 7faba54a0700 1 --2- 172.21.15.150:0/2338625064 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fab880366d0 0x7fab88038b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fab94009990 tx=0x7fab94006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:52.941 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.940+0000 7faba54a0700 1 -- 172.21.15.150:0/2338625064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faba00fc880 msgr2=0x7faba010cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:52.942 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.940+0000 7faba54a0700 1 --2- 172.21.15.150:0/2338625064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faba00fc880 0x7faba010cac0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fab90009330 tx=0x7fab90009e30 comp rx=0 tx=0).stop 2023-12-16T20:45:52.942 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.940+0000 7faba54a0700 1 -- 172.21.15.150:0/2338625064 shutdown_connections 2023-12-16T20:45:52.942 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.940+0000 7faba54a0700 1 --2- 172.21.15.150:0/2338625064 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fab880366d0 0x7fab88038b80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:52.942 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.940+0000 7faba54a0700 1 --2- 172.21.15.150:0/2338625064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faba00fc880 0x7faba010cac0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:52.942 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.940+0000 7faba54a0700 1 -- 172.21.15.150:0/2338625064 >> 172.21.15.150:0/2338625064 conn(0x7faba00f7f40 msgr2=0x7faba00f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:52.942 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.940+0000 7faba54a0700 1 -- 172.21.15.150:0/2338625064 shutdown_connections 2023-12-16T20:45:52.942 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:52.940+0000 7faba54a0700 1 -- 172.21.15.150:0/2338625064 wait complete. 2023-12-16T20:45:52.942 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:45:53.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:53 smithi052 ceph-mon[80761]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:45:53.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:53 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/2338625064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:45:54.186 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:45:54.186 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:45:54.433 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:45:54.480 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:45:55.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:54 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:45:55.117 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.114+0000 7fbcc3a7b700 1 -- 172.21.15.150:0/2972973390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcbc0fc880 msgr2=0x7fbcbc0fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:55.117 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.114+0000 7fbcc3a7b700 1 --2- 172.21.15.150:0/2972973390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcbc0fc880 0x7fbcbc0fcc90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4009c60 tx=0x7fbcb4004d10 comp rx=0 tx=0).stop 2023-12-16T20:45:55.117 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.114+0000 7fbcc3a7b700 1 -- 172.21.15.150:0/2972973390 shutdown_connections 2023-12-16T20:45:55.117 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.114+0000 7fbcc3a7b700 1 --2- 172.21.15.150:0/2972973390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcbc0fc880 0x7fbcbc0fcc90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:55.117 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.114+0000 7fbcc3a7b700 1 -- 172.21.15.150:0/2972973390 >> 172.21.15.150:0/2972973390 conn(0x7fbcbc0f7f40 msgr2=0x7fbcbc0fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:55.117 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.114+0000 7fbcc3a7b700 1 -- 172.21.15.150:0/2972973390 shutdown_connections 2023-12-16T20:45:55.117 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.114+0000 7fbcc3a7b700 1 -- 172.21.15.150:0/2972973390 wait complete. 2023-12-16T20:45:55.117 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.115+0000 7fbcc3a7b700 1 Processor -- start 2023-12-16T20:45:55.117 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.115+0000 7fbcc3a7b700 1 -- start start 2023-12-16T20:45:55.118 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.115+0000 7fbcc3a7b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcbc0fc880 0x7fbcbc071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:55.118 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.115+0000 7fbcc3a7b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc06ff90 con 0x7fbcbc0fc880 2023-12-16T20:45:55.118 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.115+0000 7fbcc1817700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcbc0fc880 0x7fbcbc071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:55.118 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.115+0000 7fbcc1817700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcbc0fc880 0x7fbcbc071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:41258/0 (socket says 172.21.15.150:41258) 2023-12-16T20:45:55.118 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.115+0000 7fbcc1817700 1 -- 172.21.15.150:0/4212061998 learned_addr learned my addr 172.21.15.150:0/4212061998 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:45:55.118 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.115+0000 7fbcc1817700 1 -- 172.21.15.150:0/4212061998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb4009940 con 0x7fbcbc0fc880 2023-12-16T20:45:55.119 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.116+0000 7fbcc1817700 1 --2- 172.21.15.150:0/4212061998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcbc0fc880 0x7fbcbc071b10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4000f80 tx=0x7fbcb4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:55.119 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.116+0000 7fbcba7fc700 1 -- 172.21.15.150:0/4212061998 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcb400bec0 con 0x7fbcbc0fc880 2023-12-16T20:45:55.119 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.116+0000 7fbcba7fc700 1 -- 172.21.15.150:0/4212061998 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcb401d8f0 con 0x7fbcbc0fc880 2023-12-16T20:45:55.119 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.116+0000 7fbcc3a7b700 1 -- 172.21.15.150:0/4212061998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcbc070190 con 0x7fbcbc0fc880 2023-12-16T20:45:55.120 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.116+0000 7fbcba7fc700 1 -- 172.21.15.150:0/4212061998 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcb4012890 con 0x7fbcbc0fc880 2023-12-16T20:45:55.120 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.116+0000 7fbcc3a7b700 1 -- 172.21.15.150:0/4212061998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcbc070680 con 0x7fbcbc0fc880 2023-12-16T20:45:55.120 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.117+0000 7fbcba7fc700 1 -- 172.21.15.150:0/4212061998 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fbcb4019040 con 0x7fbcbc0fc880 2023-12-16T20:45:55.120 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.117+0000 7fbcc3a7b700 1 -- 172.21.15.150:0/4212061998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcbc106510 con 0x7fbcbc0fc880 2023-12-16T20:45:55.120 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.117+0000 7fbcba7fc700 1 --2- 172.21.15.150:0/4212061998 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fbca40366d0 0x7fbca4038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:55.120 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.117+0000 7fbcba7fc700 1 -- 172.21.15.150:0/4212061998 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbcb404a0c0 con 0x7fbcbc0fc880 2023-12-16T20:45:55.120 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.117+0000 7fbcc1016700 1 --2- 172.21.15.150:0/4212061998 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fbca40366d0 0x7fbca4038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:55.120 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.118+0000 7fbcc1016700 1 --2- 172.21.15.150:0/4212061998 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fbca40366d0 0x7fbca4038b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009990 tx=0x7fbcb0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:55.124 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.122+0000 7fbcba7fc700 1 -- 172.21.15.150:0/4212061998 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbcb407ba00 con 0x7fbcbc0fc880 2023-12-16T20:45:55.387 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.385+0000 7fbcc3a7b700 1 -- 172.21.15.150:0/4212061998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbcbc05f270 con 0x7fbcbc0fc880 2023-12-16T20:45:55.387 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.386+0000 7fbcba7fc700 1 -- 172.21.15.150:0/4212061998 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fbcb4021020 con 0x7fbcbc0fc880 2023-12-16T20:45:55.388 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:45:55.388 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:45:55.390 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.388+0000 7fbcc3a7b700 1 -- 172.21.15.150:0/4212061998 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fbca40366d0 msgr2=0x7fbca4038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:55.390 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.388+0000 7fbcc3a7b700 1 --2- 172.21.15.150:0/4212061998 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fbca40366d0 0x7fbca4038b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009990 tx=0x7fbcb0006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:55.390 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.388+0000 7fbcc3a7b700 1 -- 172.21.15.150:0/4212061998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcbc0fc880 msgr2=0x7fbcbc071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:55.390 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.388+0000 7fbcc3a7b700 1 --2- 172.21.15.150:0/4212061998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcbc0fc880 0x7fbcbc071b10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4000f80 tx=0x7fbcb4004d10 comp rx=0 tx=0).stop 2023-12-16T20:45:55.390 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.388+0000 7fbcc3a7b700 1 -- 172.21.15.150:0/4212061998 shutdown_connections 2023-12-16T20:45:55.390 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.388+0000 7fbcc3a7b700 1 --2- 172.21.15.150:0/4212061998 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fbca40366d0 0x7fbca4038b80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:55.390 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.388+0000 7fbcc3a7b700 1 --2- 172.21.15.150:0/4212061998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcbc0fc880 0x7fbcbc071b10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:55.390 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.388+0000 7fbcc3a7b700 1 -- 172.21.15.150:0/4212061998 >> 172.21.15.150:0/4212061998 conn(0x7fbcbc0f7f40 msgr2=0x7fbcbc0f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:55.390 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.388+0000 7fbcc3a7b700 1 -- 172.21.15.150:0/4212061998 shutdown_connections 2023-12-16T20:45:55.390 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:55.388+0000 7fbcc3a7b700 1 -- 172.21.15.150:0/4212061998 wait complete. 2023-12-16T20:45:55.391 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:45:56.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:55 smithi052 ceph-mon[80761]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:45:56.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:55 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/4212061998' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:45:56.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:45:56.627 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:45:56.874 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:45:56.921 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:45:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.590+0000 7fc39d12b700 1 -- 172.21.15.150:0/3425677119 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3980fc880 msgr2=0x7fc3980fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.590+0000 7fc39d12b700 1 --2- 172.21.15.150:0/3425677119 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3980fc880 0x7fc3980fcc90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc390009c60 tx=0x7fc390004d10 comp rx=0 tx=0).stop 2023-12-16T20:45:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.590+0000 7fc39d12b700 1 -- 172.21.15.150:0/3425677119 shutdown_connections 2023-12-16T20:45:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.590+0000 7fc39d12b700 1 --2- 172.21.15.150:0/3425677119 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3980fc880 0x7fc3980fcc90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.590+0000 7fc39d12b700 1 -- 172.21.15.150:0/3425677119 >> 172.21.15.150:0/3425677119 conn(0x7fc3980f7f40 msgr2=0x7fc3980fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.590+0000 7fc39d12b700 1 -- 172.21.15.150:0/3425677119 shutdown_connections 2023-12-16T20:45:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.590+0000 7fc39d12b700 1 -- 172.21.15.150:0/3425677119 wait complete. 2023-12-16T20:45:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.591+0000 7fc39d12b700 1 Processor -- start 2023-12-16T20:45:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.591+0000 7fc39d12b700 1 -- start start 2023-12-16T20:45:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.591+0000 7fc39d12b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3980fc880 0x7fc398071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.591+0000 7fc39d12b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39806ff90 con 0x7fc3980fc880 2023-12-16T20:45:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.591+0000 7fc396d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3980fc880 0x7fc398071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.591+0000 7fc396d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3980fc880 0x7fc398071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:41276/0 (socket says 172.21.15.150:41276) 2023-12-16T20:45:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.591+0000 7fc396d9d700 1 -- 172.21.15.150:0/4233403702 learned_addr learned my addr 172.21.15.150:0/4233403702 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:45:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.591+0000 7fc396d9d700 1 -- 172.21.15.150:0/4233403702 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc390009940 con 0x7fc3980fc880 2023-12-16T20:45:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.592+0000 7fc396d9d700 1 --2- 172.21.15.150:0/4233403702 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3980fc880 0x7fc398071b10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc390015040 tx=0x7fc390004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.592+0000 7fc38f7fe700 1 -- 172.21.15.150:0/4233403702 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc39000be40 con 0x7fc3980fc880 2023-12-16T20:45:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.592+0000 7fc38f7fe700 1 -- 172.21.15.150:0/4233403702 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc39001d8f0 con 0x7fc3980fc880 2023-12-16T20:45:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.592+0000 7fc39d12b700 1 -- 172.21.15.150:0/4233403702 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc398070190 con 0x7fc3980fc880 2023-12-16T20:45:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.592+0000 7fc38f7fe700 1 -- 172.21.15.150:0/4233403702 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3900127e0 con 0x7fc3980fc880 2023-12-16T20:45:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.592+0000 7fc39d12b700 1 -- 172.21.15.150:0/4233403702 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc398070680 con 0x7fc3980fc880 2023-12-16T20:45:57.595 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.593+0000 7fc38f7fe700 1 -- 172.21.15.150:0/4233403702 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fc390019040 con 0x7fc3980fc880 2023-12-16T20:45:57.595 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.593+0000 7fc39d12b700 1 -- 172.21.15.150:0/4233403702 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc398106510 con 0x7fc3980fc880 2023-12-16T20:45:57.595 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.593+0000 7fc38f7fe700 1 --2- 172.21.15.150:0/4233403702 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fc384036720 0x7fc384038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:45:57.595 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.593+0000 7fc38f7fe700 1 -- 172.21.15.150:0/4233403702 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc39004a0e0 con 0x7fc3980fc880 2023-12-16T20:45:57.596 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.593+0000 7fc39659c700 1 --2- 172.21.15.150:0/4233403702 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fc384036720 0x7fc384038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:45:57.596 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.594+0000 7fc39659c700 1 --2- 172.21.15.150:0/4233403702 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fc384036720 0x7fc384038bd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc380009990 tx=0x7fc380006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:45:57.600 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.599+0000 7fc38f7fe700 1 -- 172.21.15.150:0/4233403702 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc39001d440 con 0x7fc3980fc880 2023-12-16T20:45:57.863 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.861+0000 7fc39d12b700 1 -- 172.21.15.150:0/4233403702 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc39804bb80 con 0x7fc3980fc880 2023-12-16T20:45:57.863 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.862+0000 7fc38f7fe700 1 -- 172.21.15.150:0/4233403702 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fc390021020 con 0x7fc3980fc880 2023-12-16T20:45:57.863 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:45:57.864 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:45:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.864+0000 7fc39d12b700 1 -- 172.21.15.150:0/4233403702 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fc384036720 msgr2=0x7fc384038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.864+0000 7fc39d12b700 1 --2- 172.21.15.150:0/4233403702 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fc384036720 0x7fc384038bd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc380009990 tx=0x7fc380006e40 comp rx=0 tx=0).stop 2023-12-16T20:45:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.864+0000 7fc39d12b700 1 -- 172.21.15.150:0/4233403702 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3980fc880 msgr2=0x7fc398071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:45:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.864+0000 7fc39d12b700 1 --2- 172.21.15.150:0/4233403702 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3980fc880 0x7fc398071b10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc390015040 tx=0x7fc390004d10 comp rx=0 tx=0).stop 2023-12-16T20:45:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.864+0000 7fc39d12b700 1 -- 172.21.15.150:0/4233403702 shutdown_connections 2023-12-16T20:45:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.864+0000 7fc39d12b700 1 --2- 172.21.15.150:0/4233403702 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fc384036720 0x7fc384038bd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.864+0000 7fc39d12b700 1 --2- 172.21.15.150:0/4233403702 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3980fc880 0x7fc398071b10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:45:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.864+0000 7fc39d12b700 1 -- 172.21.15.150:0/4233403702 >> 172.21.15.150:0/4233403702 conn(0x7fc3980f7f40 msgr2=0x7fc3980f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:45:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.864+0000 7fc39d12b700 1 -- 172.21.15.150:0/4233403702 shutdown_connections 2023-12-16T20:45:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:45:57.864+0000 7fc39d12b700 1 -- 172.21.15.150:0/4233403702 wait complete. 2023-12-16T20:45:57.867 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:45:58.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:57 smithi052 ceph-mon[80761]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:45:58.897 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:58 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/4233403702' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:45:59.128 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:45:59.129 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:45:59.376 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:45:59.423 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:45:59.692 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:45:59 smithi052 ceph-mon[80761]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:00.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.295+0000 7f462aa2b700 1 -- 172.21.15.150:0/3537168249 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46240fb1a0 msgr2=0x7f46240fd5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:00.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.295+0000 7f462aa2b700 1 --2- 172.21.15.150:0/3537168249 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46240fb1a0 0x7f46240fd5c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f46140048c0 tx=0x7f4614009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:00.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.295+0000 7f462aa2b700 1 -- 172.21.15.150:0/3537168249 shutdown_connections 2023-12-16T20:46:00.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.295+0000 7f462aa2b700 1 --2- 172.21.15.150:0/3537168249 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46240fb1a0 0x7f46240fd5c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:00.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.295+0000 7f462aa2b700 1 -- 172.21.15.150:0/3537168249 >> 172.21.15.150:0/3537168249 conn(0x7f46240f6e50 msgr2=0x7f46240f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:00.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.295+0000 7f462aa2b700 1 -- 172.21.15.150:0/3537168249 shutdown_connections 2023-12-16T20:46:00.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.295+0000 7f462aa2b700 1 -- 172.21.15.150:0/3537168249 wait complete. 2023-12-16T20:46:00.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.296+0000 7f462aa2b700 1 Processor -- start 2023-12-16T20:46:00.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.296+0000 7f462aa2b700 1 -- start start 2023-12-16T20:46:00.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.296+0000 7f462aa2b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46240fb1a0 0x7f4624113070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:00.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.296+0000 7f462aa2b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46241135b0 con 0x7f46240fb1a0 2023-12-16T20:46:00.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.296+0000 7f4623fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46240fb1a0 0x7f4624113070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:00.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.296+0000 7f4623fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46240fb1a0 0x7f4624113070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:33404/0 (socket says 172.21.15.150:33404) 2023-12-16T20:46:00.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.296+0000 7f4623fff700 1 -- 172.21.15.150:0/3003994490 learned_addr learned my addr 172.21.15.150:0/3003994490 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:46:00.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.297+0000 7f4623fff700 1 -- 172.21.15.150:0/3003994490 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4614005040 con 0x7f46240fb1a0 2023-12-16T20:46:00.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.297+0000 7f4623fff700 1 --2- 172.21.15.150:0/3003994490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46240fb1a0 0x7f4624113070 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f461400b690 tx=0x7f4614009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:00.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.297+0000 7f4620ff9700 1 -- 172.21.15.150:0/3003994490 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4614009650 con 0x7f46240fb1a0 2023-12-16T20:46:00.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.297+0000 7f4620ff9700 1 -- 172.21.15.150:0/3003994490 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46140097b0 con 0x7f46240fb1a0 2023-12-16T20:46:00.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.297+0000 7f462aa2b700 1 -- 172.21.15.150:0/3003994490 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46241137b0 con 0x7f46240fb1a0 2023-12-16T20:46:00.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.297+0000 7f462aa2b700 1 -- 172.21.15.150:0/3003994490 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4624114450 con 0x7f46240fb1a0 2023-12-16T20:46:00.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.297+0000 7f4620ff9700 1 -- 172.21.15.150:0/3003994490 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f461401c5b0 con 0x7f46240fb1a0 2023-12-16T20:46:00.302 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.298+0000 7f4620ff9700 1 -- 172.21.15.150:0/3003994490 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f461401c820 con 0x7f46240fb1a0 2023-12-16T20:46:00.302 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.298+0000 7f462aa2b700 1 -- 172.21.15.150:0/3003994490 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462405f270 con 0x7f46240fb1a0 2023-12-16T20:46:00.302 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.298+0000 7f4620ff9700 1 --2- 172.21.15.150:0/3003994490 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f460c036720 0x7f460c038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:00.303 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.298+0000 7f4620ff9700 1 -- 172.21.15.150:0/3003994490 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4614049900 con 0x7f46240fb1a0 2023-12-16T20:46:00.303 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.298+0000 7f46237fe700 1 --2- 172.21.15.150:0/3003994490 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f460c036720 0x7f460c038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:00.303 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.299+0000 7f46237fe700 1 --2- 172.21.15.150:0/3003994490 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f460c036720 0x7f460c038bd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4618009990 tx=0x7f4618006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:00.305 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.303+0000 7f4620ff9700 1 -- 172.21.15.150:0/3003994490 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4614011db0 con 0x7f46240fb1a0 2023-12-16T20:46:00.566 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.564+0000 7f462aa2b700 1 -- 172.21.15.150:0/3003994490 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4624114fa0 con 0x7f46240fb1a0 2023-12-16T20:46:00.567 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.565+0000 7f4620ff9700 1 -- 172.21.15.150:0/3003994490 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4614023990 con 0x7f46240fb1a0 2023-12-16T20:46:00.567 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:46:00.568 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:46:00.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.567+0000 7f462aa2b700 1 -- 172.21.15.150:0/3003994490 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f460c036720 msgr2=0x7f460c038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:00.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.567+0000 7f462aa2b700 1 --2- 172.21.15.150:0/3003994490 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f460c036720 0x7f460c038bd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4618009990 tx=0x7f4618006e40 comp rx=0 tx=0).stop 2023-12-16T20:46:00.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.567+0000 7f462aa2b700 1 -- 172.21.15.150:0/3003994490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46240fb1a0 msgr2=0x7f4624113070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:00.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.567+0000 7f462aa2b700 1 --2- 172.21.15.150:0/3003994490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46240fb1a0 0x7f4624113070 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f461400b690 tx=0x7f4614009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:00.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.567+0000 7f462aa2b700 1 -- 172.21.15.150:0/3003994490 shutdown_connections 2023-12-16T20:46:00.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.567+0000 7f462aa2b700 1 --2- 172.21.15.150:0/3003994490 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f460c036720 0x7f460c038bd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:00.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.567+0000 7f462aa2b700 1 --2- 172.21.15.150:0/3003994490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46240fb1a0 0x7f4624113070 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:00.571 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.567+0000 7f462aa2b700 1 -- 172.21.15.150:0/3003994490 >> 172.21.15.150:0/3003994490 conn(0x7f46240f6e50 msgr2=0x7f46240f7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:00.571 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.567+0000 7f462aa2b700 1 -- 172.21.15.150:0/3003994490 shutdown_connections 2023-12-16T20:46:00.571 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:00.567+0000 7f462aa2b700 1 -- 172.21.15.150:0/3003994490 wait complete. 2023-12-16T20:46:00.571 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:46:00.896 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:00 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/3003994490' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:01.811 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:46:01.811 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:46:02.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:01 smithi052 ceph-mon[80761]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:02.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:01 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:02.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:01 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:02.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:01 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:02.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:01 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:02.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:01 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:02.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:01 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:02.059 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:02.107 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:03.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:02 smithi052 ceph-mon[80761]: Deploying daemon prometheus.smithi052 on smithi052 2023-12-16T20:46:03.005 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.005+0000 7f929745e700 1 -- 172.21.15.150:0/2913145798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92900fc870 msgr2=0x7f92900fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:03.005 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.005+0000 7f929745e700 1 --2- 172.21.15.150:0/2913145798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92900fc870 0x7f92900fcc80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f928c0048c0 tx=0x7f928c009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:03.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.005+0000 7f929745e700 1 -- 172.21.15.150:0/2913145798 shutdown_connections 2023-12-16T20:46:03.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.005+0000 7f929745e700 1 --2- 172.21.15.150:0/2913145798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92900fc870 0x7f92900fcc80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:03.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.005+0000 7f929745e700 1 -- 172.21.15.150:0/2913145798 >> 172.21.15.150:0/2913145798 conn(0x7f92900f7f50 msgr2=0x7f92900fa360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:03.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.005+0000 7f929745e700 1 -- 172.21.15.150:0/2913145798 shutdown_connections 2023-12-16T20:46:03.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.005+0000 7f929745e700 1 -- 172.21.15.150:0/2913145798 wait complete. 2023-12-16T20:46:03.007 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.006+0000 7f929745e700 1 Processor -- start 2023-12-16T20:46:03.007 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.006+0000 7f929745e700 1 -- start start 2023-12-16T20:46:03.007 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.006+0000 7f929745e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92900fc870 0x7f9290071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:03.007 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.006+0000 7f929745e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929006ff90 con 0x7f92900fc870 2023-12-16T20:46:03.007 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.006+0000 7f92951fa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92900fc870 0x7f9290071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:03.007 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.006+0000 7f92951fa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92900fc870 0x7f9290071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:33434/0 (socket says 172.21.15.150:33434) 2023-12-16T20:46:03.007 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.006+0000 7f92951fa700 1 -- 172.21.15.150:0/2071929486 learned_addr learned my addr 172.21.15.150:0/2071929486 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:46:03.007 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.006+0000 7f92951fa700 1 -- 172.21.15.150:0/2071929486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928c005040 con 0x7f92900fc870 2023-12-16T20:46:03.007 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.007+0000 7f92951fa700 1 --2- 172.21.15.150:0/2071929486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92900fc870 0x7f9290071b10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f928c009330 tx=0x7f928c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:03.008 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.007+0000 7f9285ffb700 1 -- 172.21.15.150:0/2071929486 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f928c0096b0 con 0x7f92900fc870 2023-12-16T20:46:03.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.007+0000 7f929745e700 1 -- 172.21.15.150:0/2071929486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9290070190 con 0x7f92900fc870 2023-12-16T20:46:03.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.007+0000 7f9285ffb700 1 -- 172.21.15.150:0/2071929486 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f928c009810 con 0x7f92900fc870 2023-12-16T20:46:03.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.007+0000 7f9285ffb700 1 -- 172.21.15.150:0/2071929486 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f928c01c5b0 con 0x7f92900fc870 2023-12-16T20:46:03.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.007+0000 7f929745e700 1 -- 172.21.15.150:0/2071929486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9290070680 con 0x7f92900fc870 2023-12-16T20:46:03.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.008+0000 7f9285ffb700 1 -- 172.21.15.150:0/2071929486 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f928c013420 con 0x7f92900fc870 2023-12-16T20:46:03.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.008+0000 7f929745e700 1 -- 172.21.15.150:0/2071929486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9290106500 con 0x7f92900fc870 2023-12-16T20:46:03.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.008+0000 7f9285ffb700 1 --2- 172.21.15.150:0/2071929486 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f927c0366d0 0x7f927c038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:03.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.008+0000 7f9285ffb700 1 -- 172.21.15.150:0/2071929486 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f928c049ac0 con 0x7f92900fc870 2023-12-16T20:46:03.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.008+0000 7f92949f9700 1 --2- 172.21.15.150:0/2071929486 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f927c0366d0 0x7f927c038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:03.010 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.009+0000 7f92949f9700 1 --2- 172.21.15.150:0/2071929486 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f927c0366d0 0x7f927c038b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9280009990 tx=0x7f9280006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:03.013 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.013+0000 7f9285ffb700 1 -- 172.21.15.150:0/2071929486 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f928c0119c0 con 0x7f92900fc870 2023-12-16T20:46:03.276 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.275+0000 7f929745e700 1 -- 172.21.15.150:0/2071929486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f929005f270 con 0x7f92900fc870 2023-12-16T20:46:03.277 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.276+0000 7f9285ffb700 1 -- 172.21.15.150:0/2071929486 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f928c011370 con 0x7f92900fc870 2023-12-16T20:46:03.277 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:46:03.277 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:46:03.279 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.278+0000 7f929745e700 1 -- 172.21.15.150:0/2071929486 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f927c0366d0 msgr2=0x7f927c038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:03.279 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.278+0000 7f929745e700 1 --2- 172.21.15.150:0/2071929486 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f927c0366d0 0x7f927c038b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9280009990 tx=0x7f9280006e40 comp rx=0 tx=0).stop 2023-12-16T20:46:03.279 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.278+0000 7f929745e700 1 -- 172.21.15.150:0/2071929486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92900fc870 msgr2=0x7f9290071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:03.279 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.278+0000 7f929745e700 1 --2- 172.21.15.150:0/2071929486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92900fc870 0x7f9290071b10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f928c009330 tx=0x7f928c009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:03.279 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.278+0000 7f929745e700 1 -- 172.21.15.150:0/2071929486 shutdown_connections 2023-12-16T20:46:03.279 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.278+0000 7f929745e700 1 --2- 172.21.15.150:0/2071929486 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f927c0366d0 0x7f927c038b80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:03.279 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.278+0000 7f929745e700 1 --2- 172.21.15.150:0/2071929486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92900fc870 0x7f9290071b10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:03.279 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.278+0000 7f929745e700 1 -- 172.21.15.150:0/2071929486 >> 172.21.15.150:0/2071929486 conn(0x7f92900f7f50 msgr2=0x7f92900f8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:03.279 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.278+0000 7f929745e700 1 -- 172.21.15.150:0/2071929486 shutdown_connections 2023-12-16T20:46:03.279 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:03.278+0000 7f929745e700 1 -- 172.21.15.150:0/2071929486 wait complete. 2023-12-16T20:46:03.280 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:46:04.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:03 smithi052 ceph-mon[80761]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:04.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:03 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/2071929486' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:04.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:03 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:04.525 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:46:04.526 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:46:04.767 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:04.814 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:05.483 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.482+0000 7fa6c6d3a700 1 -- 172.21.15.150:0/765153438 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6c00fb770 msgr2=0x7fa6c00fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:05.483 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.482+0000 7fa6c6d3a700 1 --2- 172.21.15.150:0/765153438 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6c00fb770 0x7fa6c00fbb80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40048c0 tx=0x7fa6b4009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:05.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.482+0000 7fa6c6d3a700 1 -- 172.21.15.150:0/765153438 shutdown_connections 2023-12-16T20:46:05.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.482+0000 7fa6c6d3a700 1 --2- 172.21.15.150:0/765153438 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6c00fb770 0x7fa6c00fbb80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:05.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.482+0000 7fa6c6d3a700 1 -- 172.21.15.150:0/765153438 >> 172.21.15.150:0/765153438 conn(0x7fa6c00f6e50 msgr2=0x7fa6c00f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:05.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.482+0000 7fa6c6d3a700 1 -- 172.21.15.150:0/765153438 shutdown_connections 2023-12-16T20:46:05.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.482+0000 7fa6c6d3a700 1 -- 172.21.15.150:0/765153438 wait complete. 2023-12-16T20:46:05.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.482+0000 7fa6c6d3a700 1 Processor -- start 2023-12-16T20:46:05.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.482+0000 7fa6c6d3a700 1 -- start start 2023-12-16T20:46:05.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.483+0000 7fa6c6d3a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6c00fb770 0x7fa6c0114190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:05.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.483+0000 7fa6c6d3a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c01146d0 con 0x7fa6c00fb770 2023-12-16T20:46:05.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.483+0000 7fa6c4ad6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6c00fb770 0x7fa6c0114190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:05.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.483+0000 7fa6c4ad6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6c00fb770 0x7fa6c0114190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:33442/0 (socket says 172.21.15.150:33442) 2023-12-16T20:46:05.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.483+0000 7fa6c4ad6700 1 -- 172.21.15.150:0/1057794963 learned_addr learned my addr 172.21.15.150:0/1057794963 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:46:05.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.483+0000 7fa6c4ad6700 1 -- 172.21.15.150:0/1057794963 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b4005040 con 0x7fa6c00fb770 2023-12-16T20:46:05.486 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.483+0000 7fa6c4ad6700 1 --2- 172.21.15.150:0/1057794963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6c00fb770 0x7fa6c0114190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009330 tx=0x7fa6b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:05.486 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.483+0000 7fa6bd7fa700 1 -- 172.21.15.150:0/1057794963 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6b40096b0 con 0x7fa6c00fb770 2023-12-16T20:46:05.486 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.484+0000 7fa6bd7fa700 1 -- 172.21.15.150:0/1057794963 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6b4009810 con 0x7fa6c00fb770 2023-12-16T20:46:05.486 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.484+0000 7fa6c6d3a700 1 -- 172.21.15.150:0/1057794963 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c01148d0 con 0x7fa6c00fb770 2023-12-16T20:46:05.486 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.484+0000 7fa6bd7fa700 1 -- 172.21.15.150:0/1057794963 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6b401c5b0 con 0x7fa6c00fb770 2023-12-16T20:46:05.486 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.484+0000 7fa6c6d3a700 1 -- 172.21.15.150:0/1057794963 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c0115570 con 0x7fa6c00fb770 2023-12-16T20:46:05.486 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.485+0000 7fa6bd7fa700 1 -- 172.21.15.150:0/1057794963 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fa6b401c870 con 0x7fa6c00fb770 2023-12-16T20:46:05.486 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.485+0000 7fa6c6d3a700 1 -- 172.21.15.150:0/1057794963 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a40052e0 con 0x7fa6c00fb770 2023-12-16T20:46:05.487 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.485+0000 7fa6bd7fa700 1 --2- 172.21.15.150:0/1057794963 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fa6b00366d0 0x7fa6b0038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:05.487 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.485+0000 7fa6bd7fa700 1 -- 172.21.15.150:0/1057794963 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa6b4049ba0 con 0x7fa6c00fb770 2023-12-16T20:46:05.487 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.485+0000 7fa6bffff700 1 --2- 172.21.15.150:0/1057794963 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fa6b00366d0 0x7fa6b0038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:05.487 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.485+0000 7fa6bffff700 1 --2- 172.21.15.150:0/1057794963 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fa6b00366d0 0x7fa6b0038b80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac009990 tx=0x7fa6ac006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:05.494 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.493+0000 7fa6bd7fa700 1 -- 172.21.15.150:0/1057794963 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa6b4011dd0 con 0x7fa6c00fb770 2023-12-16T20:46:05.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.761+0000 7fa6c6d3a700 1 -- 172.21.15.150:0/1057794963 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6a4005590 con 0x7fa6c00fb770 2023-12-16T20:46:05.763 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.761+0000 7fa6bd7fa700 1 -- 172.21.15.150:0/1057794963 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fa6b4011370 con 0x7fa6c00fb770 2023-12-16T20:46:05.763 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:46:05.763 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:46:05.764 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.764+0000 7fa6c6d3a700 1 -- 172.21.15.150:0/1057794963 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fa6b00366d0 msgr2=0x7fa6b0038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:05.764 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.764+0000 7fa6c6d3a700 1 --2- 172.21.15.150:0/1057794963 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fa6b00366d0 0x7fa6b0038b80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac009990 tx=0x7fa6ac006e40 comp rx=0 tx=0).stop 2023-12-16T20:46:05.765 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.764+0000 7fa6c6d3a700 1 -- 172.21.15.150:0/1057794963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6c00fb770 msgr2=0x7fa6c0114190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:05.765 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.764+0000 7fa6c6d3a700 1 --2- 172.21.15.150:0/1057794963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6c00fb770 0x7fa6c0114190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009330 tx=0x7fa6b4009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:05.765 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.764+0000 7fa6c6d3a700 1 -- 172.21.15.150:0/1057794963 shutdown_connections 2023-12-16T20:46:05.765 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.764+0000 7fa6c6d3a700 1 --2- 172.21.15.150:0/1057794963 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fa6b00366d0 0x7fa6b0038b80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:05.765 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.764+0000 7fa6c6d3a700 1 --2- 172.21.15.150:0/1057794963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6c00fb770 0x7fa6c0114190 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:05.765 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.764+0000 7fa6c6d3a700 1 -- 172.21.15.150:0/1057794963 >> 172.21.15.150:0/1057794963 conn(0x7fa6c00f6e50 msgr2=0x7fa6c00f7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:05.765 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.764+0000 7fa6c6d3a700 1 -- 172.21.15.150:0/1057794963 shutdown_connections 2023-12-16T20:46:05.765 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:05.764+0000 7fa6c6d3a700 1 -- 172.21.15.150:0/1057794963 wait complete. 2023-12-16T20:46:05.765 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:46:06.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:05 smithi052 ceph-mon[80761]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:07.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:06 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/1057794963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:07.012 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:46:07.012 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:46:07.250 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:07.296 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:07.912 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:07 smithi052 ceph-mon[80761]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:08.330 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.329+0000 7fb38b567700 1 -- 172.21.15.150:0/1521667015 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3840659e0 msgr2=0x7fb3840ff470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:08.331 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.329+0000 7fb38b567700 1 --2- 172.21.15.150:0/1521667015 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3840659e0 0x7fb3840ff470 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0048c0 tx=0x7fb37c009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:08.331 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.329+0000 7fb38b567700 1 -- 172.21.15.150:0/1521667015 shutdown_connections 2023-12-16T20:46:08.331 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.329+0000 7fb38b567700 1 --2- 172.21.15.150:0/1521667015 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3840659e0 0x7fb3840ff470 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:08.331 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.329+0000 7fb38b567700 1 -- 172.21.15.150:0/1521667015 >> 172.21.15.150:0/1521667015 conn(0x7fb3840f6e80 msgr2=0x7fb3840f9290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:08.331 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.329+0000 7fb38b567700 1 -- 172.21.15.150:0/1521667015 shutdown_connections 2023-12-16T20:46:08.331 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.329+0000 7fb38b567700 1 -- 172.21.15.150:0/1521667015 wait complete. 2023-12-16T20:46:08.331 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.330+0000 7fb38b567700 1 Processor -- start 2023-12-16T20:46:08.331 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.330+0000 7fb38b567700 1 -- start start 2023-12-16T20:46:08.331 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.330+0000 7fb38b567700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3840659e0 0x7fb38410ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:08.331 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.330+0000 7fb38b567700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38410cf60 con 0x7fb3840659e0 2023-12-16T20:46:08.332 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.330+0000 7fb389303700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3840659e0 0x7fb38410ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:08.332 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.330+0000 7fb389303700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3840659e0 0x7fb38410ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:33458/0 (socket says 172.21.15.150:33458) 2023-12-16T20:46:08.332 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.331+0000 7fb389303700 1 -- 172.21.15.150:0/831128018 learned_addr learned my addr 172.21.15.150:0/831128018 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:46:08.332 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.331+0000 7fb389303700 1 -- 172.21.15.150:0/831128018 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37c005040 con 0x7fb3840659e0 2023-12-16T20:46:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.331+0000 7fb389303700 1 --2- 172.21.15.150:0/831128018 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3840659e0 0x7fb38410ca20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb37c009330 tx=0x7fb37c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.331+0000 7fb379ffb700 1 -- 172.21.15.150:0/831128018 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb37c0096b0 con 0x7fb3840659e0 2023-12-16T20:46:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.331+0000 7fb379ffb700 1 -- 172.21.15.150:0/831128018 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb37c009810 con 0x7fb3840659e0 2023-12-16T20:46:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.331+0000 7fb379ffb700 1 -- 172.21.15.150:0/831128018 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb37c01c5b0 con 0x7fb3840659e0 2023-12-16T20:46:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.331+0000 7fb38b567700 1 -- 172.21.15.150:0/831128018 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38410d160 con 0x7fb3840659e0 2023-12-16T20:46:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.332+0000 7fb38b567700 1 -- 172.21.15.150:0/831128018 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38410de00 con 0x7fb3840659e0 2023-12-16T20:46:08.334 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.332+0000 7fb38b567700 1 -- 172.21.15.150:0/831128018 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb38405f270 con 0x7fb3840659e0 2023-12-16T20:46:08.334 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.332+0000 7fb379ffb700 1 -- 172.21.15.150:0/831128018 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fb37c013420 con 0x7fb3840659e0 2023-12-16T20:46:08.334 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.333+0000 7fb379ffb700 1 --2- 172.21.15.150:0/831128018 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fb3700366e0 0x7fb370038b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:08.334 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.333+0000 7fb379ffb700 1 -- 172.21.15.150:0/831128018 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb37c049a90 con 0x7fb3840659e0 2023-12-16T20:46:08.334 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.333+0000 7fb388b02700 1 --2- 172.21.15.150:0/831128018 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fb3700366e0 0x7fb370038b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:08.334 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.333+0000 7fb388b02700 1 --2- 172.21.15.150:0/831128018 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fb3700366e0 0x7fb370038b90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb374009990 tx=0x7fb374006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:08.338 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.338+0000 7fb379ffb700 1 -- 172.21.15.150:0/831128018 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb37c0119c0 con 0x7fb3840659e0 2023-12-16T20:46:08.595 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.595+0000 7fb38b567700 1 -- 172.21.15.150:0/831128018 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb38410e950 con 0x7fb3840659e0 2023-12-16T20:46:08.596 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.596+0000 7fb379ffb700 1 -- 172.21.15.150:0/831128018 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb37c011370 con 0x7fb3840659e0 2023-12-16T20:46:08.596 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:46:08.596 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:46:08.598 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.597+0000 7fb38b567700 1 -- 172.21.15.150:0/831128018 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fb3700366e0 msgr2=0x7fb370038b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:08.598 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.597+0000 7fb38b567700 1 --2- 172.21.15.150:0/831128018 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fb3700366e0 0x7fb370038b90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb374009990 tx=0x7fb374006e40 comp rx=0 tx=0).stop 2023-12-16T20:46:08.598 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.597+0000 7fb38b567700 1 -- 172.21.15.150:0/831128018 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3840659e0 msgr2=0x7fb38410ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:08.598 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.597+0000 7fb38b567700 1 --2- 172.21.15.150:0/831128018 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3840659e0 0x7fb38410ca20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb37c009330 tx=0x7fb37c009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:08.598 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.598+0000 7fb38b567700 1 -- 172.21.15.150:0/831128018 shutdown_connections 2023-12-16T20:46:08.599 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.598+0000 7fb38b567700 1 --2- 172.21.15.150:0/831128018 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fb3700366e0 0x7fb370038b90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:08.599 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.598+0000 7fb38b567700 1 --2- 172.21.15.150:0/831128018 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3840659e0 0x7fb38410ca20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:08.599 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.598+0000 7fb38b567700 1 -- 172.21.15.150:0/831128018 >> 172.21.15.150:0/831128018 conn(0x7fb3840f6e80 msgr2=0x7fb3840f7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:08.599 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.598+0000 7fb38b567700 1 -- 172.21.15.150:0/831128018 shutdown_connections 2023-12-16T20:46:08.599 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:08.598+0000 7fb38b567700 1 -- 172.21.15.150:0/831128018 wait complete. 2023-12-16T20:46:08.599 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:46:08.800 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:08 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/831128018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:09.833 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:09 smithi052 ceph-mon[80761]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:09.843 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:46:09.843 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:46:10.086 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:10.132 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:10.822 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.821+0000 7f39555d6700 1 -- 172.21.15.150:0/38406487 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39500fb780 msgr2=0x7f39500fbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:10.822 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.821+0000 7f39555d6700 1 --2- 172.21.15.150:0/38406487 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39500fb780 0x7f39500fbb90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3940009c60 tx=0x7f3940004d10 comp rx=0 tx=0).stop 2023-12-16T20:46:10.822 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.821+0000 7f39555d6700 1 -- 172.21.15.150:0/38406487 shutdown_connections 2023-12-16T20:46:10.822 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.821+0000 7f39555d6700 1 --2- 172.21.15.150:0/38406487 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39500fb780 0x7f39500fbb90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:10.822 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.821+0000 7f39555d6700 1 -- 172.21.15.150:0/38406487 >> 172.21.15.150:0/38406487 conn(0x7f39500f6e40 msgr2=0x7f39500f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:10.822 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.821+0000 7f39555d6700 1 -- 172.21.15.150:0/38406487 shutdown_connections 2023-12-16T20:46:10.822 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.821+0000 7f39555d6700 1 -- 172.21.15.150:0/38406487 wait complete. 2023-12-16T20:46:10.824 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.822+0000 7f39555d6700 1 Processor -- start 2023-12-16T20:46:10.824 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.822+0000 7f39555d6700 1 -- start start 2023-12-16T20:46:10.824 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.822+0000 7f39555d6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39500fb780 0x7f395010ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:10.824 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.822+0000 7f39555d6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395010bf80 con 0x7f39500fb780 2023-12-16T20:46:10.824 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.822+0000 7f394effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39500fb780 0x7f395010ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:10.824 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.822+0000 7f394effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39500fb780 0x7f395010ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:48368/0 (socket says 172.21.15.150:48368) 2023-12-16T20:46:10.824 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.822+0000 7f394effd700 1 -- 172.21.15.150:0/2541689618 learned_addr learned my addr 172.21.15.150:0/2541689618 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:46:10.828 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.823+0000 7f394effd700 1 -- 172.21.15.150:0/2541689618 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3940009940 con 0x7f39500fb780 2023-12-16T20:46:10.828 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.823+0000 7f394effd700 1 --2- 172.21.15.150:0/2541689618 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39500fb780 0x7f395010ba40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3940015040 tx=0x7f3940004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:10.828 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.823+0000 7f39377fe700 1 -- 172.21.15.150:0/2541689618 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f394000be40 con 0x7f39500fb780 2023-12-16T20:46:10.828 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.823+0000 7f39555d6700 1 -- 172.21.15.150:0/2541689618 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f395010c180 con 0x7f39500fb780 2023-12-16T20:46:10.828 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.823+0000 7f39377fe700 1 -- 172.21.15.150:0/2541689618 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f394001d8f0 con 0x7f39500fb780 2023-12-16T20:46:10.828 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.823+0000 7f39377fe700 1 -- 172.21.15.150:0/2541689618 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39400127e0 con 0x7f39500fb780 2023-12-16T20:46:10.828 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.823+0000 7f39555d6700 1 -- 172.21.15.150:0/2541689618 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395010ce20 con 0x7f39500fb780 2023-12-16T20:46:10.829 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.824+0000 7f39377fe700 1 -- 172.21.15.150:0/2541689618 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f3940019040 con 0x7f39500fb780 2023-12-16T20:46:10.829 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.824+0000 7f39555d6700 1 -- 172.21.15.150:0/2541689618 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f395005f270 con 0x7f39500fb780 2023-12-16T20:46:10.829 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.824+0000 7f39377fe700 1 --2- 172.21.15.150:0/2541689618 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f39380366d0 0x7f3938038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:10.829 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.824+0000 7f39377fe700 1 -- 172.21.15.150:0/2541689618 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f394004a0f0 con 0x7f39500fb780 2023-12-16T20:46:10.829 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.824+0000 7f394e7fc700 1 --2- 172.21.15.150:0/2541689618 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f39380366d0 0x7f3938038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:10.829 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.825+0000 7f394e7fc700 1 --2- 172.21.15.150:0/2541689618 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f39380366d0 0x7f3938038b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3944009990 tx=0x7f3944006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:10.830 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:10.829+0000 7f39377fe700 1 -- 172.21.15.150:0/2541689618 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f394001d440 con 0x7f39500fb780 2023-12-16T20:46:11.053 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.052+0000 7f39377fe700 1 -- 172.21.15.150:0/2541689618 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 14) v1 ==== 43351+0+0 (secure 0 0 0) 0x7f394002e350 con 0x7f39500fb780 2023-12-16T20:46:11.080 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.076+0000 7f394e7fc700 1 -- 172.21.15.150:0/2541689618 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f39380366d0 msgr2=0x7f3938038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-12-16T20:46:11.080 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.076+0000 7f394e7fc700 1 -- 172.21.15.150:0/2541689618 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f39380366d0 msgr2=0x7f3938038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-12-16T20:46:11.080 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.076+0000 7f394e7fc700 1 --2- 172.21.15.150:0/2541689618 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f39380366d0 0x7f3938038b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3944009990 tx=0x7f3944006e40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-16T20:46:11.080 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.076+0000 7f394e7fc700 1 --2- 172.21.15.150:0/2541689618 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f39380366d0 0x7f3938038b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3944009990 tx=0x7f3944006e40 comp rx=0 tx=0).stop 2023-12-16T20:46:11.080 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.076+0000 7f39377fe700 1 -- 172.21.15.150:0/2541689618 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f39380366d0 msgr2=0x7f3938038b80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-12-16T20:46:11.080 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.076+0000 7f39377fe700 1 --2- 172.21.15.150:0/2541689618 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f39380366d0 0x7f3938038b80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:11.098 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.097+0000 7f39555d6700 1 -- 172.21.15.150:0/2541689618 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3930005160 con 0x7f39500fb780 2023-12-16T20:46:11.098 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.098+0000 7f39377fe700 1 -- 172.21.15.150:0/2541689618 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f3940021300 con 0x7f39500fb780 2023-12-16T20:46:11.098 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:46:11.098 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:46:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.100+0000 7f39555d6700 1 -- 172.21.15.150:0/2541689618 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39500fb780 msgr2=0x7f395010ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.100+0000 7f39555d6700 1 --2- 172.21.15.150:0/2541689618 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39500fb780 0x7f395010ba40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3940015040 tx=0x7f3940004d10 comp rx=0 tx=0).stop 2023-12-16T20:46:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.100+0000 7f39555d6700 1 -- 172.21.15.150:0/2541689618 shutdown_connections 2023-12-16T20:46:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.100+0000 7f39555d6700 1 --2- 172.21.15.150:0/2541689618 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7f39380366d0 0x7f3938038b80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.100+0000 7f39555d6700 1 --2- 172.21.15.150:0/2541689618 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39500fb780 0x7f395010ba40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.100+0000 7f39555d6700 1 -- 172.21.15.150:0/2541689618 >> 172.21.15.150:0/2541689618 conn(0x7f39500f6e40 msgr2=0x7f39500f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.100+0000 7f39555d6700 1 -- 172.21.15.150:0/2541689618 shutdown_connections 2023-12-16T20:46:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:11.100+0000 7f39555d6700 1 -- 172.21.15.150:0/2541689618 wait complete. 2023-12-16T20:46:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:46:11.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:10 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:11.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:10 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:11.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:10 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:11.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:10 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-16T20:46:11.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:10 smithi052 ceph-mon[80761]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:12.342 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:46:12.342 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:46:12.355 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:12 smithi052 ceph-mon[80761]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-16T20:46:12.356 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:12 smithi052 ceph-mon[80761]: mgrmap e14: smithi052.nmkcbs(active, since 42s) 2023-12-16T20:46:12.356 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:12 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/2541689618' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:12.590 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:12.637 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:13.280 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.278+0000 7fe4116d9700 1 -- 172.21.15.150:0/1506850543 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe40c0fb770 msgr2=0x7fe40c0fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:13.280 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.278+0000 7fe4116d9700 1 --2- 172.21.15.150:0/1506850543 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe40c0fb770 0x7fe40c0fbb80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe404009c60 tx=0x7fe404004d10 comp rx=0 tx=0).stop 2023-12-16T20:46:13.280 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.278+0000 7fe4116d9700 1 -- 172.21.15.150:0/1506850543 shutdown_connections 2023-12-16T20:46:13.280 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.278+0000 7fe4116d9700 1 --2- 172.21.15.150:0/1506850543 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe40c0fb770 0x7fe40c0fbb80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:13.280 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.278+0000 7fe4116d9700 1 -- 172.21.15.150:0/1506850543 >> 172.21.15.150:0/1506850543 conn(0x7fe40c0f6e50 msgr2=0x7fe40c0f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:13.280 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.278+0000 7fe4116d9700 1 -- 172.21.15.150:0/1506850543 shutdown_connections 2023-12-16T20:46:13.280 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.278+0000 7fe4116d9700 1 -- 172.21.15.150:0/1506850543 wait complete. 2023-12-16T20:46:13.280 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.279+0000 7fe4116d9700 1 Processor -- start 2023-12-16T20:46:13.280 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.279+0000 7fe4116d9700 1 -- start start 2023-12-16T20:46:13.281 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.279+0000 7fe4116d9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe40c0fb770 0x7fe40c10cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:13.281 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.279+0000 7fe4116d9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c10d0e0 con 0x7fe40c0fb770 2023-12-16T20:46:13.281 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.279+0000 7fe40affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe40c0fb770 0x7fe40c10cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:13.281 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.279+0000 7fe40affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe40c0fb770 0x7fe40c10cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:48392/0 (socket says 172.21.15.150:48392) 2023-12-16T20:46:13.281 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.279+0000 7fe40affd700 1 -- 172.21.15.150:0/3563673387 learned_addr learned my addr 172.21.15.150:0/3563673387 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:46:13.281 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.280+0000 7fe40affd700 1 -- 172.21.15.150:0/3563673387 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe404009940 con 0x7fe40c0fb770 2023-12-16T20:46:13.282 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.280+0000 7fe40affd700 1 --2- 172.21.15.150:0/3563673387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe40c0fb770 0x7fe40c10cba0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe404000f80 tx=0x7fe404004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:13.282 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.280+0000 7fe3f37fe700 1 -- 172.21.15.150:0/3563673387 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe40400bec0 con 0x7fe40c0fb770 2023-12-16T20:46:13.282 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.280+0000 7fe3f37fe700 1 -- 172.21.15.150:0/3563673387 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe40401d8f0 con 0x7fe40c0fb770 2023-12-16T20:46:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.280+0000 7fe4116d9700 1 -- 172.21.15.150:0/3563673387 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40c10d2e0 con 0x7fe40c0fb770 2023-12-16T20:46:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.280+0000 7fe3f37fe700 1 -- 172.21.15.150:0/3563673387 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe404012890 con 0x7fe40c0fb770 2023-12-16T20:46:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.280+0000 7fe4116d9700 1 -- 172.21.15.150:0/3563673387 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40c10df80 con 0x7fe40c0fb770 2023-12-16T20:46:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.281+0000 7fe3f37fe700 1 -- 172.21.15.150:0/3563673387 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 14) v1 ==== 43351+0+0 (secure 0 0 0) 0x7fe404019040 con 0x7fe40c0fb770 2023-12-16T20:46:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.281+0000 7fe3f37fe700 1 --2- 172.21.15.150:0/3563673387 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fe3f4036720 0x7fe3f4038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.281+0000 7fe4116d9700 1 -- 172.21.15.150:0/3563673387 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40c05f270 con 0x7fe40c0fb770 2023-12-16T20:46:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.281+0000 7fe3f37fe700 1 -- 172.21.15.150:0/3563673387 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe40404a290 con 0x7fe40c0fb770 2023-12-16T20:46:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.281+0000 7fe40a7fc700 1 -- 172.21.15.150:0/3563673387 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fe3f4036720 msgr2=0x7fe3f4038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2799692744 2023-12-16T20:46:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.281+0000 7fe40a7fc700 1 --2- 172.21.15.150:0/3563673387 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fe3f4036720 0x7fe3f4038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T20:46:13.286 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.286+0000 7fe3f37fe700 1 -- 172.21.15.150:0/3563673387 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe404021020 con 0x7fe40c0fb770 2023-12-16T20:46:13.482 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.482+0000 7fe40a7fc700 1 -- 172.21.15.150:0/3563673387 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fe3f4036720 msgr2=0x7fe3f4038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2799692744 2023-12-16T20:46:13.483 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.482+0000 7fe40a7fc700 1 --2- 172.21.15.150:0/3563673387 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fe3f4036720 0x7fe3f4038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T20:46:13.544 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.544+0000 7fe4116d9700 1 -- 172.21.15.150:0/3563673387 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe40c10e270 con 0x7fe40c0fb770 2023-12-16T20:46:13.545 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.544+0000 7fe3f37fe700 1 -- 172.21.15.150:0/3563673387 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fe404035090 con 0x7fe40c0fb770 2023-12-16T20:46:13.545 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:46:13.545 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:46:13.547 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.546+0000 7fe4116d9700 1 -- 172.21.15.150:0/3563673387 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fe3f4036720 msgr2=0x7fe3f4038bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T20:46:13.547 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.546+0000 7fe4116d9700 1 --2- 172.21.15.150:0/3563673387 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fe3f4036720 0x7fe3f4038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:13.547 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.546+0000 7fe4116d9700 1 -- 172.21.15.150:0/3563673387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe40c0fb770 msgr2=0x7fe40c10cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:13.547 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.546+0000 7fe4116d9700 1 --2- 172.21.15.150:0/3563673387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe40c0fb770 0x7fe40c10cba0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe404000f80 tx=0x7fe404004d10 comp rx=0 tx=0).stop 2023-12-16T20:46:13.547 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.546+0000 7fe4116d9700 1 -- 172.21.15.150:0/3563673387 shutdown_connections 2023-12-16T20:46:13.547 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.546+0000 7fe4116d9700 1 --2- 172.21.15.150:0/3563673387 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fe3f4036720 0x7fe3f4038bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:13.547 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.546+0000 7fe4116d9700 1 --2- 172.21.15.150:0/3563673387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe40c0fb770 0x7fe40c10cba0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:13.547 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.546+0000 7fe4116d9700 1 -- 172.21.15.150:0/3563673387 >> 172.21.15.150:0/3563673387 conn(0x7fe40c0f6e50 msgr2=0x7fe40c0f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:13.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.546+0000 7fe4116d9700 1 -- 172.21.15.150:0/3563673387 shutdown_connections 2023-12-16T20:46:13.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:13.546+0000 7fe4116d9700 1 -- 172.21.15.150:0/3563673387 wait complete. 2023-12-16T20:46:13.548 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:46:14.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:13 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/3563673387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:14.791 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:46:14.791 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:46:15.037 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:15.084 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:15.947 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.946+0000 7fef8d493700 1 -- 172.21.15.150:0/255279819 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef880fb760 msgr2=0x7fef880fbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:15.947 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.946+0000 7fef8d493700 1 --2- 172.21.15.150:0/255279819 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef880fb760 0x7fef880fbb70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fef780048c0 tx=0x7fef78009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:15.947 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.946+0000 7fef8d493700 1 -- 172.21.15.150:0/255279819 shutdown_connections 2023-12-16T20:46:15.947 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.946+0000 7fef8d493700 1 --2- 172.21.15.150:0/255279819 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef880fb760 0x7fef880fbb70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:15.947 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.946+0000 7fef8d493700 1 -- 172.21.15.150:0/255279819 >> 172.21.15.150:0/255279819 conn(0x7fef880f6ec0 msgr2=0x7fef880f92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:15.947 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.946+0000 7fef8d493700 1 -- 172.21.15.150:0/255279819 shutdown_connections 2023-12-16T20:46:15.947 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.946+0000 7fef8d493700 1 -- 172.21.15.150:0/255279819 wait complete. 2023-12-16T20:46:15.948 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.947+0000 7fef8d493700 1 Processor -- start 2023-12-16T20:46:15.948 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.947+0000 7fef8d493700 1 -- start start 2023-12-16T20:46:15.949 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.947+0000 7fef8d493700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef880fb760 0x7fef8810f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:15.949 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.947+0000 7fef8d493700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef88066460 con 0x7fef880fb760 2023-12-16T20:46:15.949 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.947+0000 7fef86ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef880fb760 0x7fef8810f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:15.949 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.947+0000 7fef86ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef880fb760 0x7fef8810f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:48406/0 (socket says 172.21.15.150:48406) 2023-12-16T20:46:15.949 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.947+0000 7fef86ffd700 1 -- 172.21.15.150:0/707234105 learned_addr learned my addr 172.21.15.150:0/707234105 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:46:15.949 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.947+0000 7fef86ffd700 1 -- 172.21.15.150:0/707234105 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef78005040 con 0x7fef880fb760 2023-12-16T20:46:15.949 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.948+0000 7fef86ffd700 1 --2- 172.21.15.150:0/707234105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef880fb760 0x7fef8810f770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fef7800b690 tx=0x7fef78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:15.949 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.948+0000 7fef7ffff700 1 -- 172.21.15.150:0/707234105 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef78009650 con 0x7fef880fb760 2023-12-16T20:46:15.949 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.948+0000 7fef8d493700 1 -- 172.21.15.150:0/707234105 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef88066660 con 0x7fef880fb760 2023-12-16T20:46:15.949 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.948+0000 7fef7ffff700 1 -- 172.21.15.150:0/707234105 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef780097b0 con 0x7fef880fb760 2023-12-16T20:46:15.949 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.948+0000 7fef7ffff700 1 -- 172.21.15.150:0/707234105 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef7801c5b0 con 0x7fef880fb760 2023-12-16T20:46:15.949 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.948+0000 7fef8d493700 1 -- 172.21.15.150:0/707234105 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef8810fef0 con 0x7fef880fb760 2023-12-16T20:46:15.950 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.949+0000 7fef7ffff700 1 -- 172.21.15.150:0/707234105 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 14) v1 ==== 43351+0+0 (secure 0 0 0) 0x7fef78013420 con 0x7fef880fb760 2023-12-16T20:46:15.950 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.949+0000 7fef8d493700 1 -- 172.21.15.150:0/707234105 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef88105370 con 0x7fef880fb760 2023-12-16T20:46:15.950 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.949+0000 7fef7ffff700 1 --2- 172.21.15.150:0/707234105 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fef74036720 0x7fef74038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:15.950 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.949+0000 7fef7ffff700 1 -- 172.21.15.150:0/707234105 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fef78049a40 con 0x7fef880fb760 2023-12-16T20:46:15.951 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.949+0000 7fef7e7fc700 1 -- 172.21.15.150:0/707234105 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fef74036720 msgr2=0x7fef74038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2799692744 2023-12-16T20:46:15.951 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.949+0000 7fef7e7fc700 1 --2- 172.21.15.150:0/707234105 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fef74036720 0x7fef74038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T20:46:15.954 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:15.954+0000 7fef7ffff700 1 -- 172.21.15.150:0/707234105 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fef78025400 con 0x7fef880fb760 2023-12-16T20:46:16.150 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:16.150+0000 7fef7e7fc700 1 -- 172.21.15.150:0/707234105 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fef74036720 msgr2=0x7fef74038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2799692744 2023-12-16T20:46:16.151 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:16.150+0000 7fef7e7fc700 1 --2- 172.21.15.150:0/707234105 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fef74036720 0x7fef74038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T20:46:16.222 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:16.222+0000 7fef8d493700 1 -- 172.21.15.150:0/707234105 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef8804bb80 con 0x7fef880fb760 2023-12-16T20:46:16.222 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:16.222+0000 7fef7ffff700 1 -- 172.21.15.150:0/707234105 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fef78011370 con 0x7fef880fb760 2023-12-16T20:46:16.223 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:46:16.223 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:46:16.224 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:16.224+0000 7fef8d493700 1 -- 172.21.15.150:0/707234105 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fef74036720 msgr2=0x7fef74038bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T20:46:16.225 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:16.224+0000 7fef8d493700 1 --2- 172.21.15.150:0/707234105 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fef74036720 0x7fef74038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:16.225 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:16.224+0000 7fef8d493700 1 -- 172.21.15.150:0/707234105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef880fb760 msgr2=0x7fef8810f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:16.225 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:16.224+0000 7fef8d493700 1 --2- 172.21.15.150:0/707234105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef880fb760 0x7fef8810f770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fef7800b690 tx=0x7fef78009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:16.225 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:16.224+0000 7fef8d493700 1 -- 172.21.15.150:0/707234105 shutdown_connections 2023-12-16T20:46:16.225 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:16.224+0000 7fef8d493700 1 --2- 172.21.15.150:0/707234105 >> [v2:172.21.15.52:6800/2799692744,v1:172.21.15.52:6801/2799692744] conn(0x7fef74036720 0x7fef74038bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:16.225 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:16.224+0000 7fef8d493700 1 --2- 172.21.15.150:0/707234105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef880fb760 0x7fef8810f770 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:16.225 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:16.224+0000 7fef8d493700 1 -- 172.21.15.150:0/707234105 >> 172.21.15.150:0/707234105 conn(0x7fef880f6ec0 msgr2=0x7fef880f7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:16.225 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:16.224+0000 7fef8d493700 1 -- 172.21.15.150:0/707234105 shutdown_connections 2023-12-16T20:46:16.225 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:16.224+0000 7fef8d493700 1 -- 172.21.15.150:0/707234105 wait complete. 2023-12-16T20:46:16.226 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:46:16.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:16 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/707234105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:17.515 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:46:17.516 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:46:17.763 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:17.808 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:18.546 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.545+0000 7f4487b84700 1 -- 172.21.15.150:0/1001805358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44800fb780 msgr2=0x7f44800fbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:18.546 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.545+0000 7f4487b84700 1 --2- 172.21.15.150:0/1001805358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44800fb780 0x7f44800fbb90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f447c0048c0 tx=0x7f447c009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:18.546 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.545+0000 7f4487b84700 1 -- 172.21.15.150:0/1001805358 shutdown_connections 2023-12-16T20:46:18.546 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.545+0000 7f4487b84700 1 --2- 172.21.15.150:0/1001805358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44800fb780 0x7f44800fbb90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:18.546 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.545+0000 7f4487b84700 1 -- 172.21.15.150:0/1001805358 >> 172.21.15.150:0/1001805358 conn(0x7f44800f6e40 msgr2=0x7f44800f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:18.546 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.545+0000 7f4487b84700 1 -- 172.21.15.150:0/1001805358 shutdown_connections 2023-12-16T20:46:18.546 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.545+0000 7f4487b84700 1 -- 172.21.15.150:0/1001805358 wait complete. 2023-12-16T20:46:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.546+0000 7f4487b84700 1 Processor -- start 2023-12-16T20:46:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.546+0000 7f4487b84700 1 -- start start 2023-12-16T20:46:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.546+0000 7f4487b84700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44800fb780 0x7f448010cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.546+0000 7f4487b84700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448010cff0 con 0x7f44800fb780 2023-12-16T20:46:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.546+0000 7f4485920700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44800fb780 0x7f448010cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.546+0000 7f4485920700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44800fb780 0x7f448010cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:58730/0 (socket says 172.21.15.150:58730) 2023-12-16T20:46:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.546+0000 7f4485920700 1 -- 172.21.15.150:0/1711737673 learned_addr learned my addr 172.21.15.150:0/1711737673 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:46:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.546+0000 7f4485920700 1 -- 172.21.15.150:0/1711737673 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c005040 con 0x7f44800fb780 2023-12-16T20:46:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.547+0000 7f4485920700 1 --2- 172.21.15.150:0/1711737673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44800fb780 0x7f448010cab0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f447c009330 tx=0x7f447c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.547+0000 7f44767fc700 1 -- 172.21.15.150:0/1711737673 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f447c0096b0 con 0x7f44800fb780 2023-12-16T20:46:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.547+0000 7f44767fc700 1 -- 172.21.15.150:0/1711737673 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f447c009810 con 0x7f44800fb780 2023-12-16T20:46:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.547+0000 7f4487b84700 1 -- 172.21.15.150:0/1711737673 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f448010d1f0 con 0x7f44800fb780 2023-12-16T20:46:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.547+0000 7f44767fc700 1 -- 172.21.15.150:0/1711737673 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f447c01c5b0 con 0x7f44800fb780 2023-12-16T20:46:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.547+0000 7f4487b84700 1 -- 172.21.15.150:0/1711737673 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f448010de90 con 0x7f44800fb780 2023-12-16T20:46:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.548+0000 7f44767fc700 1 -- 172.21.15.150:0/1711737673 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 15) v1 ==== 43075+0+0 (secure 0 0 0) 0x7f447c013420 con 0x7f44800fb780 2023-12-16T20:46:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.548+0000 7f4487b84700 1 -- 172.21.15.150:0/1711737673 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f448005f270 con 0x7f44800fb780 2023-12-16T20:46:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.548+0000 7f44767fc700 1 -- 172.21.15.150:0/1711737673 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f447c0496b0 con 0x7f44800fb780 2023-12-16T20:46:18.553 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.553+0000 7f44767fc700 1 -- 172.21.15.150:0/1711737673 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f447c0119c0 con 0x7f44800fb780 2023-12-16T20:46:18.665 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: Active manager daemon smithi052.nmkcbs restarted 2023-12-16T20:46:18.665 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: Activating manager daemon smithi052.nmkcbs 2023-12-16T20:46:18.665 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: osdmap e5: 0 total, 0 up, 0 in 2023-12-16T20:46:18.665 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: mgrmap e15: smithi052.nmkcbs(active, starting, since 0.00294665s) 2023-12-16T20:46:18.665 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2023-12-16T20:46:18.665 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr metadata", "who": "smithi052.nmkcbs", "id": "smithi052.nmkcbs"}]: dispatch 2023-12-16T20:46:18.666 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T20:46:18.666 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T20:46:18.666 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T20:46:18.666 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: Manager daemon smithi052.nmkcbs is now available 2023-12-16T20:46:18.666 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:46:18.666 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:46:18.666 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:18.666 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:46:18.666 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.nmkcbs/mirror_snapshot_schedule"}]: dispatch 2023-12-16T20:46:18.666 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:18 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.nmkcbs/trash_purge_schedule"}]: dispatch 2023-12-16T20:46:18.838 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.837+0000 7f4487b84700 1 -- 172.21.15.150:0/1711737673 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f448010e210 con 0x7f44800fb780 2023-12-16T20:46:18.839 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.838+0000 7f44767fc700 1 -- 172.21.15.150:0/1711737673 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f447c011370 con 0x7f44800fb780 2023-12-16T20:46:18.839 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:46:18.839 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:46:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.841+0000 7f4487b84700 1 -- 172.21.15.150:0/1711737673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44800fb780 msgr2=0x7f448010cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.841+0000 7f4487b84700 1 --2- 172.21.15.150:0/1711737673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44800fb780 0x7f448010cab0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f447c009330 tx=0x7f447c009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:18.842 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.841+0000 7f4487b84700 1 -- 172.21.15.150:0/1711737673 shutdown_connections 2023-12-16T20:46:18.842 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.841+0000 7f4487b84700 1 --2- 172.21.15.150:0/1711737673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44800fb780 0x7f448010cab0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:18.842 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.841+0000 7f4487b84700 1 -- 172.21.15.150:0/1711737673 >> 172.21.15.150:0/1711737673 conn(0x7f44800f6e40 msgr2=0x7f44800f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:18.842 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.841+0000 7f4487b84700 1 -- 172.21.15.150:0/1711737673 shutdown_connections 2023-12-16T20:46:18.842 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:18.841+0000 7f4487b84700 1 -- 172.21.15.150:0/1711737673 wait complete. 2023-12-16T20:46:18.842 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:46:19.694 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:19 smithi052 ceph-mon[80761]: [16/Dec/2023:20:46:18] ENGINE Bus STARTING 2023-12-16T20:46:19.695 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:19 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:19.695 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:19 smithi052 ceph-mon[80761]: [16/Dec/2023:20:46:18] ENGINE Serving on https://172.21.15.52:7150 2023-12-16T20:46:19.695 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:19 smithi052 ceph-mon[80761]: [16/Dec/2023:20:46:18] ENGINE Bus STARTED 2023-12-16T20:46:19.695 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:19 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:19.695 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:19 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:19.695 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:19 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/1711737673' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:19.695 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:19 smithi052 ceph-mon[80761]: mgrmap e16: smithi052.nmkcbs(active, since 1.00476s) 2023-12-16T20:46:20.091 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:46:20.091 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:46:20.454 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:20.507 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T20:46:20.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:20 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:21.303 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.301+0000 7f5143d7d700 1 -- 172.21.15.150:0/1218636991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f513c0fab50 msgr2=0x7f513c0faf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:21.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.301+0000 7f5143d7d700 1 --2- 172.21.15.150:0/1218636991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f513c0fab50 0x7f513c0faf60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5134009c60 tx=0x7f5134004d10 comp rx=0 tx=0).stop 2023-12-16T20:46:21.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.301+0000 7f5143d7d700 1 -- 172.21.15.150:0/1218636991 shutdown_connections 2023-12-16T20:46:21.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.301+0000 7f5143d7d700 1 --2- 172.21.15.150:0/1218636991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f513c0fab50 0x7f513c0faf60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:21.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.301+0000 7f5143d7d700 1 -- 172.21.15.150:0/1218636991 >> 172.21.15.150:0/1218636991 conn(0x7f513c0f65d0 msgr2=0x7f513c0f8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:21.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.301+0000 7f5143d7d700 1 -- 172.21.15.150:0/1218636991 shutdown_connections 2023-12-16T20:46:21.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.301+0000 7f5143d7d700 1 -- 172.21.15.150:0/1218636991 wait complete. 2023-12-16T20:46:21.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.302+0000 7f5143d7d700 1 Processor -- start 2023-12-16T20:46:21.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.302+0000 7f5143d7d700 1 -- start start 2023-12-16T20:46:21.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.302+0000 7f5143d7d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f513c0fab50 0x7f513c0fd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:21.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.302+0000 7f5143d7d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c0fbaf0 con 0x7f513c0fab50 2023-12-16T20:46:21.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.302+0000 7f5141b19700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f513c0fab50 0x7f513c0fd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:21.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.302+0000 7f5141b19700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f513c0fab50 0x7f513c0fd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:58762/0 (socket says 172.21.15.150:58762) 2023-12-16T20:46:21.305 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.302+0000 7f5141b19700 1 -- 172.21.15.150:0/1529287890 learned_addr learned my addr 172.21.15.150:0/1529287890 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:46:21.305 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.303+0000 7f5141b19700 1 -- 172.21.15.150:0/1529287890 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5134009940 con 0x7f513c0fab50 2023-12-16T20:46:21.305 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.303+0000 7f5141b19700 1 --2- 172.21.15.150:0/1529287890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f513c0fab50 0x7f513c0fd670 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5134000f80 tx=0x7f5134004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:21.305 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.303+0000 7f51327fc700 1 -- 172.21.15.150:0/1529287890 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f513400be90 con 0x7f513c0fab50 2023-12-16T20:46:21.305 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.303+0000 7f5143d7d700 1 -- 172.21.15.150:0/1529287890 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513c0fbcf0 con 0x7f513c0fab50 2023-12-16T20:46:21.305 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.303+0000 7f51327fc700 1 -- 172.21.15.150:0/1529287890 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f513401d8f0 con 0x7f513c0fab50 2023-12-16T20:46:21.305 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.303+0000 7f51327fc700 1 -- 172.21.15.150:0/1529287890 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5134012810 con 0x7f513c0fab50 2023-12-16T20:46:21.305 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.303+0000 7f5143d7d700 1 -- 172.21.15.150:0/1529287890 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513c0fc1e0 con 0x7f513c0fab50 2023-12-16T20:46:21.306 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.304+0000 7f51327fc700 1 -- 172.21.15.150:0/1529287890 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f5134019040 con 0x7f513c0fab50 2023-12-16T20:46:21.306 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.304+0000 7f5143d7d700 1 -- 172.21.15.150:0/1529287890 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513c1064c0 con 0x7f513c0fab50 2023-12-16T20:46:21.306 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.304+0000 7f51327fc700 1 --2- 172.21.15.150:0/1529287890 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f51280367c0 0x7f5128038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:21.306 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.304+0000 7f51327fc700 1 -- 172.21.15.150:0/1529287890 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f513404a240 con 0x7f513c0fab50 2023-12-16T20:46:21.306 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.304+0000 7f5141318700 1 --2- 172.21.15.150:0/1529287890 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f51280367c0 0x7f5128038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:21.307 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.305+0000 7f5141318700 1 --2- 172.21.15.150:0/1529287890 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f51280367c0 0x7f5128038c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f512c009990 tx=0x7f512c006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:21.309 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.309+0000 7f51327fc700 1 -- 172.21.15.150:0/1529287890 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f513401d440 con 0x7f513c0fab50 2023-12-16T20:46:21.574 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.573+0000 7f5143d7d700 1 -- 172.21.15.150:0/1529287890 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f513c05f270 con 0x7f513c0fab50 2023-12-16T20:46:21.574 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.574+0000 7f51327fc700 1 -- 172.21.15.150:0/1529287890 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f5134021090 con 0x7f513c0fab50 2023-12-16T20:46:21.575 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:46:21.575 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:46:21.577 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.576+0000 7f5143d7d700 1 -- 172.21.15.150:0/1529287890 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f51280367c0 msgr2=0x7f5128038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:21.577 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.576+0000 7f5143d7d700 1 --2- 172.21.15.150:0/1529287890 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f51280367c0 0x7f5128038c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f512c009990 tx=0x7f512c006e40 comp rx=0 tx=0).stop 2023-12-16T20:46:21.577 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.576+0000 7f5143d7d700 1 -- 172.21.15.150:0/1529287890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f513c0fab50 msgr2=0x7f513c0fd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:21.577 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.576+0000 7f5143d7d700 1 --2- 172.21.15.150:0/1529287890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f513c0fab50 0x7f513c0fd670 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5134000f80 tx=0x7f5134004d10 comp rx=0 tx=0).stop 2023-12-16T20:46:21.577 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.576+0000 7f5143d7d700 1 -- 172.21.15.150:0/1529287890 shutdown_connections 2023-12-16T20:46:21.577 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.576+0000 7f5143d7d700 1 --2- 172.21.15.150:0/1529287890 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f51280367c0 0x7f5128038c70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:21.577 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.576+0000 7f5143d7d700 1 --2- 172.21.15.150:0/1529287890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f513c0fab50 0x7f513c0fd670 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:21.577 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.576+0000 7f5143d7d700 1 -- 172.21.15.150:0/1529287890 >> 172.21.15.150:0/1529287890 conn(0x7f513c0f65d0 msgr2=0x7f513c0f74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:21.577 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.576+0000 7f5143d7d700 1 -- 172.21.15.150:0/1529287890 shutdown_connections 2023-12-16T20:46:21.577 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:21.576+0000 7f5143d7d700 1 -- 172.21.15.150:0/1529287890 wait complete. 2023-12-16T20:46:21.578 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:46:22.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:21 smithi052 ceph-mon[80761]: mgrmap e17: smithi052.nmkcbs(active, since 2s) 2023-12-16T20:46:22.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:22.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:22.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:22.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:22.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:22.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:22.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:46:22.817 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:46:22.817 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:46:23.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:22 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/1529287890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:22 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:22 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:22 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:46:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:22 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:22 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:46:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:22 smithi052 ceph-mon[80761]: Updating smithi052:/etc/ceph/ceph.conf 2023-12-16T20:46:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:22 smithi052 ceph-mon[80761]: Updating smithi150:/etc/ceph/ceph.conf 2023-12-16T20:46:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:22 smithi052 ceph-mon[80761]: Updating smithi150:/var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T20:46:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:22 smithi052 ceph-mon[80761]: Updating smithi052:/var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T20:46:23.070 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T20:46:24.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:23 smithi052 ceph-mon[80761]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2023-12-16T20:46:24.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:23 smithi052 ceph-mon[80761]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2023-12-16T20:46:24.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:23 smithi052 ceph-mon[80761]: Updating smithi150:/var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T20:46:24.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:23 smithi052 ceph-mon[80761]: Updating smithi052:/var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T20:46:24.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:24.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:24.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:24.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:24.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:24.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-16T20:46:24.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-16T20:46:24.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:24.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:23 smithi052 ceph-mon[80761]: Deploying daemon crash.smithi150 on smithi150 2023-12-16T20:46:24.425 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.421+0000 7f8649de7700 1 -- 172.21.15.150:0/1389967940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86441028f0 msgr2=0x7f8644102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:24.425 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.421+0000 7f8649de7700 1 --2- 172.21.15.150:0/1389967940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86441028f0 0x7f8644102cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f86340048c0 tx=0x7f8634009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:24.425 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.422+0000 7f8649de7700 1 -- 172.21.15.150:0/1389967940 shutdown_connections 2023-12-16T20:46:24.425 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.422+0000 7f8649de7700 1 --2- 172.21.15.150:0/1389967940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86441028f0 0x7f8644102cc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:24.425 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.422+0000 7f8649de7700 1 -- 172.21.15.150:0/1389967940 >> 172.21.15.150:0/1389967940 conn(0x7f86440715e0 msgr2=0x7f86440719e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:24.425 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.422+0000 7f8649de7700 1 -- 172.21.15.150:0/1389967940 shutdown_connections 2023-12-16T20:46:24.425 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.422+0000 7f8649de7700 1 -- 172.21.15.150:0/1389967940 wait complete. 2023-12-16T20:46:24.425 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.422+0000 7f8649de7700 1 Processor -- start 2023-12-16T20:46:24.425 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.423+0000 7f8649de7700 1 -- start start 2023-12-16T20:46:24.425 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.423+0000 7f8649de7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86441028f0 0x7f8644112b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:24.425 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.423+0000 7f8649de7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8644113090 con 0x7f86441028f0 2023-12-16T20:46:24.426 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.423+0000 7f86437fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86441028f0 0x7f8644112b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:24.426 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.423+0000 7f86437fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86441028f0 0x7f8644112b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:58780/0 (socket says 172.21.15.150:58780) 2023-12-16T20:46:24.426 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.423+0000 7f86437fe700 1 -- 172.21.15.150:0/3984074714 learned_addr learned my addr 172.21.15.150:0/3984074714 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:46:24.426 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.423+0000 7f86437fe700 1 -- 172.21.15.150:0/3984074714 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8634005040 con 0x7f86441028f0 2023-12-16T20:46:24.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.424+0000 7f86437fe700 1 --2- 172.21.15.150:0/3984074714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86441028f0 0x7f8644112b50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8634009300 tx=0x7f8634009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:24.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.424+0000 7f862bfff700 1 -- 172.21.15.150:0/3984074714 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8634004e60 con 0x7f86441028f0 2023-12-16T20:46:24.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.424+0000 7f862bfff700 1 -- 172.21.15.150:0/3984074714 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8634013420 con 0x7f86441028f0 2023-12-16T20:46:24.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.424+0000 7f8649de7700 1 -- 172.21.15.150:0/3984074714 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8644113290 con 0x7f86441028f0 2023-12-16T20:46:24.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.424+0000 7f862bfff700 1 -- 172.21.15.150:0/3984074714 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f863401c5b0 con 0x7f86441028f0 2023-12-16T20:46:24.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.424+0000 7f8649de7700 1 -- 172.21.15.150:0/3984074714 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864410ca10 con 0x7f86441028f0 2023-12-16T20:46:24.428 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.425+0000 7f862bfff700 1 -- 172.21.15.150:0/3984074714 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f863401c7f0 con 0x7f86441028f0 2023-12-16T20:46:24.428 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.425+0000 7f862bfff700 1 --2- 172.21.15.150:0/3984074714 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f862c0367c0 0x7f862c038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:24.428 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.425+0000 7f862bfff700 1 -- 172.21.15.150:0/3984074714 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f8634049a50 con 0x7f86441028f0 2023-12-16T20:46:24.428 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.425+0000 7f8649de7700 1 -- 172.21.15.150:0/3984074714 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864405f270 con 0x7f86441028f0 2023-12-16T20:46:24.428 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.425+0000 7f8642ffd700 1 --2- 172.21.15.150:0/3984074714 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f862c0367c0 0x7f862c038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:24.428 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.426+0000 7f8642ffd700 1 --2- 172.21.15.150:0/3984074714 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f862c0367c0 0x7f862c038c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8638009990 tx=0x7f8638006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:24.432 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.430+0000 7f862bfff700 1 -- 172.21.15.150:0/3984074714 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8634023300 con 0x7f86441028f0 2023-12-16T20:46:24.724 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.722+0000 7f8649de7700 1 -- 172.21.15.150:0/3984074714 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f864404bb80 con 0x7f86441028f0 2023-12-16T20:46:24.724 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.723+0000 7f862bfff700 1 -- 172.21.15.150:0/3984074714 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f8634034090 con 0x7f86441028f0 2023-12-16T20:46:24.725 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:46:24.725 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:46:24.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.725+0000 7f8649de7700 1 -- 172.21.15.150:0/3984074714 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f862c0367c0 msgr2=0x7f862c038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:24.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.725+0000 7f8649de7700 1 --2- 172.21.15.150:0/3984074714 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f862c0367c0 0x7f862c038c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8638009990 tx=0x7f8638006e40 comp rx=0 tx=0).stop 2023-12-16T20:46:24.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.725+0000 7f8649de7700 1 -- 172.21.15.150:0/3984074714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86441028f0 msgr2=0x7f8644112b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:24.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.725+0000 7f8649de7700 1 --2- 172.21.15.150:0/3984074714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86441028f0 0x7f8644112b50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8634009300 tx=0x7f8634009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:24.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.725+0000 7f8649de7700 1 -- 172.21.15.150:0/3984074714 shutdown_connections 2023-12-16T20:46:24.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.725+0000 7f8649de7700 1 --2- 172.21.15.150:0/3984074714 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f862c0367c0 0x7f862c038c70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:24.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.725+0000 7f8649de7700 1 --2- 172.21.15.150:0/3984074714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86441028f0 0x7f8644112b50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:24.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.725+0000 7f8649de7700 1 -- 172.21.15.150:0/3984074714 >> 172.21.15.150:0/3984074714 conn(0x7f86440715e0 msgr2=0x7f8644103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:24.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.725+0000 7f8649de7700 1 -- 172.21.15.150:0/3984074714 shutdown_connections 2023-12-16T20:46:24.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:24.725+0000 7f8649de7700 1 -- 172.21.15.150:0/3984074714 wait complete. 2023-12-16T20:46:24.728 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:46:25.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:24 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/3984074714' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:26.000 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:46:26.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:46:26.282 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T20:46:26.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:26.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:26.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:26.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:26.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:26 smithi052 ceph-mon[80761]: Deploying daemon node-exporter.smithi150 on smithi150 2023-12-16T20:46:27.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.127+0000 7f3c9bdcd700 1 -- 172.21.15.150:0/3408593085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c941017d0 msgr2=0x7f3c94101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:27.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.127+0000 7f3c9bdcd700 1 --2- 172.21.15.150:0/3408593085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c941017d0 0x7f3c94101ba0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3c900048c0 tx=0x7f3c90009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:27.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.127+0000 7f3c9bdcd700 1 -- 172.21.15.150:0/3408593085 shutdown_connections 2023-12-16T20:46:27.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.127+0000 7f3c9bdcd700 1 --2- 172.21.15.150:0/3408593085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c941017d0 0x7f3c94101ba0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:27.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.127+0000 7f3c9bdcd700 1 -- 172.21.15.150:0/3408593085 >> 172.21.15.150:0/3408593085 conn(0x7f3c940f7310 msgr2=0x7f3c940f9720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:27.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.127+0000 7f3c9bdcd700 1 -- 172.21.15.150:0/3408593085 shutdown_connections 2023-12-16T20:46:27.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.127+0000 7f3c9bdcd700 1 -- 172.21.15.150:0/3408593085 wait complete. 2023-12-16T20:46:27.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.128+0000 7f3c9bdcd700 1 Processor -- start 2023-12-16T20:46:27.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.128+0000 7f3c9bdcd700 1 -- start start 2023-12-16T20:46:27.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.128+0000 7f3c9bdcd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c941017d0 0x7f3c9410dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:27.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.128+0000 7f3c9bdcd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9410e150 con 0x7f3c941017d0 2023-12-16T20:46:27.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.128+0000 7f3c99b69700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c941017d0 0x7f3c9410dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:27.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.128+0000 7f3c99b69700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c941017d0 0x7f3c9410dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:58810/0 (socket says 172.21.15.150:58810) 2023-12-16T20:46:27.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.128+0000 7f3c99b69700 1 -- 172.21.15.150:0/2053791766 learned_addr learned my addr 172.21.15.150:0/2053791766 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:46:27.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.128+0000 7f3c99b69700 1 -- 172.21.15.150:0/2053791766 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c90005040 con 0x7f3c941017d0 2023-12-16T20:46:27.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.129+0000 7f3c99b69700 1 --2- 172.21.15.150:0/2053791766 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c941017d0 0x7f3c9410dc10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009300 tx=0x7f3c90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:27.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.129+0000 7f3c8a7fc700 1 -- 172.21.15.150:0/2053791766 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c90004e60 con 0x7f3c941017d0 2023-12-16T20:46:27.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.129+0000 7f3c9bdcd700 1 -- 172.21.15.150:0/2053791766 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9410e350 con 0x7f3c941017d0 2023-12-16T20:46:27.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.129+0000 7f3c8a7fc700 1 -- 172.21.15.150:0/2053791766 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c90013420 con 0x7f3c941017d0 2023-12-16T20:46:27.132 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.129+0000 7f3c8a7fc700 1 -- 172.21.15.150:0/2053791766 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c9001c5b0 con 0x7f3c941017d0 2023-12-16T20:46:27.132 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.129+0000 7f3c9bdcd700 1 -- 172.21.15.150:0/2053791766 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9410e840 con 0x7f3c941017d0 2023-12-16T20:46:27.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.130+0000 7f3c8a7fc700 1 -- 172.21.15.150:0/2053791766 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f3c90009600 con 0x7f3c941017d0 2023-12-16T20:46:27.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.130+0000 7f3c8a7fc700 1 --2- 172.21.15.150:0/2053791766 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f3c80036810 0x7f3c80038cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:27.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.130+0000 7f3c8a7fc700 1 -- 172.21.15.150:0/2053791766 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f3c90049130 con 0x7f3c941017d0 2023-12-16T20:46:27.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.130+0000 7f3c99368700 1 --2- 172.21.15.150:0/2053791766 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f3c80036810 0x7f3c80038cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:27.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.130+0000 7f3c9bdcd700 1 -- 172.21.15.150:0/2053791766 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c94100a40 con 0x7f3c941017d0 2023-12-16T20:46:27.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.131+0000 7f3c99368700 1 --2- 172.21.15.150:0/2053791766 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f3c80036810 0x7f3c80038cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3c84009990 tx=0x7f3c84006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:27.137 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.136+0000 7f3c8a7fc700 1 -- 172.21.15.150:0/2053791766 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3c90013590 con 0x7f3c941017d0 2023-12-16T20:46:27.436 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.434+0000 7f3c9bdcd700 1 -- 172.21.15.150:0/2053791766 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c9404bb80 con 0x7f3c941017d0 2023-12-16T20:46:27.437 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.434+0000 7f3c8a7fc700 1 -- 172.21.15.150:0/2053791766 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f3c90021210 con 0x7f3c941017d0 2023-12-16T20:46:27.437 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:46:27.437 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:46:27.438 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.437+0000 7f3c9bdcd700 1 -- 172.21.15.150:0/2053791766 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f3c80036810 msgr2=0x7f3c80038cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:27.438 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.437+0000 7f3c9bdcd700 1 --2- 172.21.15.150:0/2053791766 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f3c80036810 0x7f3c80038cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3c84009990 tx=0x7f3c84006e40 comp rx=0 tx=0).stop 2023-12-16T20:46:27.438 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.437+0000 7f3c9bdcd700 1 -- 172.21.15.150:0/2053791766 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c941017d0 msgr2=0x7f3c9410dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:27.438 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.437+0000 7f3c9bdcd700 1 --2- 172.21.15.150:0/2053791766 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c941017d0 0x7f3c9410dc10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009300 tx=0x7f3c90009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:27.438 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.437+0000 7f3c9bdcd700 1 -- 172.21.15.150:0/2053791766 shutdown_connections 2023-12-16T20:46:27.439 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.437+0000 7f3c9bdcd700 1 --2- 172.21.15.150:0/2053791766 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f3c80036810 0x7f3c80038cc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:27.439 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.437+0000 7f3c9bdcd700 1 --2- 172.21.15.150:0/2053791766 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c941017d0 0x7f3c9410dc10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:27.439 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.437+0000 7f3c9bdcd700 1 -- 172.21.15.150:0/2053791766 >> 172.21.15.150:0/2053791766 conn(0x7f3c940f7310 msgr2=0x7f3c940f92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:27.439 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.437+0000 7f3c9bdcd700 1 -- 172.21.15.150:0/2053791766 shutdown_connections 2023-12-16T20:46:27.439 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:27.437+0000 7f3c9bdcd700 1 -- 172.21.15.150:0/2053791766 wait complete. 2023-12-16T20:46:27.439 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:46:28.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:28 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/2053791766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:28.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:28 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:29.231 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:46:29.232 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:46:29.499 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T20:46:30.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:29 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:30.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:29 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:30.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:29 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:30.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:29 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:30.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:29 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vjiupe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T20:46:30.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:29 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vjiupe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-12-16T20:46:30.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:29 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T20:46:30.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:29 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:30.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:29 smithi052 ceph-mon[80761]: Deploying daemon mgr.smithi150.vjiupe on smithi150 2023-12-16T20:46:30.232 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.228+0000 7f96c3ab5700 1 -- 172.21.15.150:0/3370970314 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96bc1017e0 msgr2=0x7f96bc101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:30.232 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.228+0000 7f96c3ab5700 1 --2- 172.21.15.150:0/3370970314 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96bc1017e0 0x7f96bc101bb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f96b40048c0 tx=0x7f96b4009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:30.232 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.228+0000 7f96c3ab5700 1 -- 172.21.15.150:0/3370970314 shutdown_connections 2023-12-16T20:46:30.232 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.228+0000 7f96c3ab5700 1 --2- 172.21.15.150:0/3370970314 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96bc1017e0 0x7f96bc101bb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:30.232 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.228+0000 7f96c3ab5700 1 -- 172.21.15.150:0/3370970314 >> 172.21.15.150:0/3370970314 conn(0x7f96bc0715e0 msgr2=0x7f96bc0719e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:30.232 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.228+0000 7f96c3ab5700 1 -- 172.21.15.150:0/3370970314 shutdown_connections 2023-12-16T20:46:30.232 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.228+0000 7f96c3ab5700 1 -- 172.21.15.150:0/3370970314 wait complete. 2023-12-16T20:46:30.233 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.228+0000 7f96c3ab5700 1 Processor -- start 2023-12-16T20:46:30.233 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.228+0000 7f96c3ab5700 1 -- start start 2023-12-16T20:46:30.233 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.228+0000 7f96c3ab5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96bc1017e0 0x7f96bc111a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:30.233 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.228+0000 7f96c3ab5700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc111f70 con 0x7f96bc1017e0 2023-12-16T20:46:30.233 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.229+0000 7f96c1851700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96bc1017e0 0x7f96bc111a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:30.233 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.229+0000 7f96c1851700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96bc1017e0 0x7f96bc111a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:45058/0 (socket says 172.21.15.150:45058) 2023-12-16T20:46:30.233 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.229+0000 7f96c1851700 1 -- 172.21.15.150:0/2523335709 learned_addr learned my addr 172.21.15.150:0/2523335709 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:46:30.233 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.229+0000 7f96c1851700 1 -- 172.21.15.150:0/2523335709 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b4005040 con 0x7f96bc1017e0 2023-12-16T20:46:30.233 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.229+0000 7f96c1851700 1 --2- 172.21.15.150:0/2523335709 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96bc1017e0 0x7f96bc111a30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f96b4009300 tx=0x7f96b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:30.233 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.229+0000 7f96b27fc700 1 -- 172.21.15.150:0/2523335709 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96b4004e60 con 0x7f96bc1017e0 2023-12-16T20:46:30.233 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.229+0000 7f96b27fc700 1 -- 172.21.15.150:0/2523335709 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96b4013420 con 0x7f96bc1017e0 2023-12-16T20:46:30.234 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.229+0000 7f96c3ab5700 1 -- 172.21.15.150:0/2523335709 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96bc112170 con 0x7f96bc1017e0 2023-12-16T20:46:30.234 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.229+0000 7f96b27fc700 1 -- 172.21.15.150:0/2523335709 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96b401c5b0 con 0x7f96bc1017e0 2023-12-16T20:46:30.234 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.229+0000 7f96c3ab5700 1 -- 172.21.15.150:0/2523335709 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96bc10b8f0 con 0x7f96bc1017e0 2023-12-16T20:46:30.234 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.230+0000 7f96c3ab5700 1 -- 172.21.15.150:0/2523335709 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96bc05f270 con 0x7f96bc1017e0 2023-12-16T20:46:30.234 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.230+0000 7f96b27fc700 1 -- 172.21.15.150:0/2523335709 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f96b4009600 con 0x7f96bc1017e0 2023-12-16T20:46:30.234 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.230+0000 7f96b27fc700 1 --2- 172.21.15.150:0/2523335709 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f96a80367c0 0x7f96a8038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:30.234 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.231+0000 7f96b27fc700 1 -- 172.21.15.150:0/2523335709 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f96b4049a00 con 0x7f96bc1017e0 2023-12-16T20:46:30.234 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.231+0000 7f96c1050700 1 --2- 172.21.15.150:0/2523335709 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f96a80367c0 0x7f96a8038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:30.235 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.231+0000 7f96c1050700 1 --2- 172.21.15.150:0/2523335709 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f96a80367c0 0x7f96a8038c70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009990 tx=0x7f96ac006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:30.237 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.236+0000 7f96b27fc700 1 -- 172.21.15.150:0/2523335709 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f96b4025400 con 0x7f96bc1017e0 2023-12-16T20:46:30.531 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.529+0000 7f96c3ab5700 1 -- 172.21.15.150:0/2523335709 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96bc10d010 con 0x7f96bc1017e0 2023-12-16T20:46:30.531 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.529+0000 7f96b27fc700 1 -- 172.21.15.150:0/2523335709 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f96b4011370 con 0x7f96bc1017e0 2023-12-16T20:46:30.531 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:46:30.531 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:44:38.105612Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T20:46:30.534 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.532+0000 7f96c3ab5700 1 -- 172.21.15.150:0/2523335709 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f96a80367c0 msgr2=0x7f96a8038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:30.534 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.532+0000 7f96c3ab5700 1 --2- 172.21.15.150:0/2523335709 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f96a80367c0 0x7f96a8038c70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009990 tx=0x7f96ac006e40 comp rx=0 tx=0).stop 2023-12-16T20:46:30.534 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.532+0000 7f96c3ab5700 1 -- 172.21.15.150:0/2523335709 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96bc1017e0 msgr2=0x7f96bc111a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:30.534 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.532+0000 7f96c3ab5700 1 --2- 172.21.15.150:0/2523335709 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96bc1017e0 0x7f96bc111a30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f96b4009300 tx=0x7f96b4009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:30.534 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.533+0000 7f96c3ab5700 1 -- 172.21.15.150:0/2523335709 shutdown_connections 2023-12-16T20:46:30.534 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.533+0000 7f96c3ab5700 1 --2- 172.21.15.150:0/2523335709 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f96a80367c0 0x7f96a8038c70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:30.535 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.533+0000 7f96c3ab5700 1 --2- 172.21.15.150:0/2523335709 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96bc1017e0 0x7f96bc111a30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:30.535 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.533+0000 7f96c3ab5700 1 -- 172.21.15.150:0/2523335709 >> 172.21.15.150:0/2523335709 conn(0x7f96bc0715e0 msgr2=0x7f96bc102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:30.535 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.533+0000 7f96c3ab5700 1 -- 172.21.15.150:0/2523335709 shutdown_connections 2023-12-16T20:46:30.535 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:30.533+0000 7f96c3ab5700 1 -- 172.21.15.150:0/2523335709 wait complete. 2023-12-16T20:46:30.535 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-12-16T20:46:31.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:31.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:31.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:31.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:31 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/2523335709' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:31.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:31.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T20:46:31.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:31.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:31 smithi052 ceph-mon[80761]: Deploying daemon mon.smithi150 on smithi150 2023-12-16T20:46:31.792 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T20:46:31.793 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T20:46:32.073 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi150/config 2023-12-16T20:46:32.933 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.929+0000 7fc50081d700 1 -- 172.21.15.150:0/3117385685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4fc0fd9e0 msgr2=0x7fc4fc0fddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:32.933 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.930+0000 7fc50081d700 1 --2- 172.21.15.150:0/3117385685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4fc0fd9e0 0x7fc4fc0fddb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0048c0 tx=0x7fc4ec009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:32.933 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.930+0000 7fc50081d700 1 -- 172.21.15.150:0/3117385685 shutdown_connections 2023-12-16T20:46:32.933 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.930+0000 7fc50081d700 1 --2- 172.21.15.150:0/3117385685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4fc0fd9e0 0x7fc4fc0fddb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:32.933 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.930+0000 7fc50081d700 1 -- 172.21.15.150:0/3117385685 >> 172.21.15.150:0/3117385685 conn(0x7fc4fc0f7410 msgr2=0x7fc4fc0f9820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:32.933 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.930+0000 7fc50081d700 1 -- 172.21.15.150:0/3117385685 shutdown_connections 2023-12-16T20:46:32.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.930+0000 7fc50081d700 1 -- 172.21.15.150:0/3117385685 wait complete. 2023-12-16T20:46:32.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.930+0000 7fc50081d700 1 Processor -- start 2023-12-16T20:46:32.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.930+0000 7fc50081d700 1 -- start start 2023-12-16T20:46:32.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.930+0000 7fc50081d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4fc0fd9e0 0x7fc4fc112d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:32.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.931+0000 7fc50081d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc113260 con 0x7fc4fc0fd9e0 2023-12-16T20:46:32.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.931+0000 7fc4fa59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4fc0fd9e0 0x7fc4fc112d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:32.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.931+0000 7fc4fa59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4fc0fd9e0 0x7fc4fc112d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.150:45086/0 (socket says 172.21.15.150:45086) 2023-12-16T20:46:32.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.931+0000 7fc4fa59c700 1 -- 172.21.15.150:0/1307711935 learned_addr learned my addr 172.21.15.150:0/1307711935 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:46:32.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.931+0000 7fc4fa59c700 1 -- 172.21.15.150:0/1307711935 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec005040 con 0x7fc4fc0fd9e0 2023-12-16T20:46:32.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.931+0000 7fc4fa59c700 1 --2- 172.21.15.150:0/1307711935 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4fc0fd9e0 0x7fc4fc112d20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec009300 tx=0x7fc4ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:32.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.931+0000 7fc4eaffd700 1 -- 172.21.15.150:0/1307711935 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4ec004e60 con 0x7fc4fc0fd9e0 2023-12-16T20:46:32.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.932+0000 7fc4eaffd700 1 -- 172.21.15.150:0/1307711935 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4ec013420 con 0x7fc4fc0fd9e0 2023-12-16T20:46:32.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.932+0000 7fc50081d700 1 -- 172.21.15.150:0/1307711935 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4fc113460 con 0x7fc4fc0fd9e0 2023-12-16T20:46:32.934 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.932+0000 7fc50081d700 1 -- 172.21.15.150:0/1307711935 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4fc1138d0 con 0x7fc4fc0fd9e0 2023-12-16T20:46:32.935 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.932+0000 7fc4eaffd700 1 -- 172.21.15.150:0/1307711935 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4ec01c5b0 con 0x7fc4fc0fd9e0 2023-12-16T20:46:32.935 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.933+0000 7fc4eaffd700 1 -- 172.21.15.150:0/1307711935 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7fc4ec01c7a0 con 0x7fc4fc0fd9e0 2023-12-16T20:46:32.935 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.933+0000 7fc50081d700 1 -- 172.21.15.150:0/1307711935 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4fc05f270 con 0x7fc4fc0fd9e0 2023-12-16T20:46:32.936 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.933+0000 7fc4eaffd700 1 --2- 172.21.15.150:0/1307711935 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc4e4036820 0x7fc4e4038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:32.936 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.933+0000 7fc4eaffd700 1 -- 172.21.15.150:0/1307711935 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fc4ec04a990 con 0x7fc4fc0fd9e0 2023-12-16T20:46:32.936 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.933+0000 7fc4f9d9b700 1 --2- 172.21.15.150:0/1307711935 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc4e4036820 0x7fc4e4038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:32.936 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.934+0000 7fc4f9d9b700 1 --2- 172.21.15.150:0/1307711935 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc4e4036820 0x7fc4e4038cd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0009990 tx=0x7fc4f0006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:32.940 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.938+0000 7fc4eaffd700 1 -- 172.21.15.150:0/1307711935 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc4ec011db0 con 0x7fc4fc0fd9e0 2023-12-16T20:46:32.979 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/4233403702' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:32.987 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:32.988 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/3003994490' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:32.988 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:32.988 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.988 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.988 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.988 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.988 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.988 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.988 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Deploying daemon prometheus.smithi052 on smithi052 2023-12-16T20:46:32.988 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:32.988 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/2071929486' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:32.988 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.988 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:32.989 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/1057794963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:32.989 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:32.989 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/831128018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:32.989 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:32.989 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.989 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.989 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.989 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-16T20:46:32.989 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:32.989 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14162 172.21.15.52:0/3712385098' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-16T20:46:32.989 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: mgrmap e14: smithi052.nmkcbs(active, since 42s) 2023-12-16T20:46:32.989 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/2541689618' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:32.989 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/3563673387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:32.989 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/707234105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:32.990 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Active manager daemon smithi052.nmkcbs restarted 2023-12-16T20:46:32.990 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Activating manager daemon smithi052.nmkcbs 2023-12-16T20:46:32.990 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: osdmap e5: 0 total, 0 up, 0 in 2023-12-16T20:46:32.990 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: mgrmap e15: smithi052.nmkcbs(active, starting, since 0.00294665s) 2023-12-16T20:46:32.990 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2023-12-16T20:46:32.990 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr metadata", "who": "smithi052.nmkcbs", "id": "smithi052.nmkcbs"}]: dispatch 2023-12-16T20:46:32.990 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T20:46:32.990 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T20:46:32.990 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T20:46:32.990 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Manager daemon smithi052.nmkcbs is now available 2023-12-16T20:46:32.990 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:46:32.990 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:46:32.990 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.991 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:46:32.991 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.nmkcbs/mirror_snapshot_schedule"}]: dispatch 2023-12-16T20:46:32.991 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.nmkcbs/trash_purge_schedule"}]: dispatch 2023-12-16T20:46:32.991 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: [16/Dec/2023:20:46:18] ENGINE Bus STARTING 2023-12-16T20:46:32.991 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.991 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: [16/Dec/2023:20:46:18] ENGINE Serving on https://172.21.15.52:7150 2023-12-16T20:46:32.991 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: [16/Dec/2023:20:46:18] ENGINE Bus STARTED 2023-12-16T20:46:32.991 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.991 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.991 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/1711737673' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:32.991 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: mgrmap e16: smithi052.nmkcbs(active, since 1.00476s) 2023-12-16T20:46:32.991 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.991 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: mgrmap e17: smithi052.nmkcbs(active, since 2s) 2023-12-16T20:46:32.991 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.991 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:46:32.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/1529287890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:32.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:46:32.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:32.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:46:32.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Updating smithi052:/etc/ceph/ceph.conf 2023-12-16T20:46:32.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Updating smithi150:/etc/ceph/ceph.conf 2023-12-16T20:46:32.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Updating smithi150:/var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T20:46:32.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Updating smithi052:/var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T20:46:32.993 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2023-12-16T20:46:32.993 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2023-12-16T20:46:32.993 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Updating smithi150:/var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T20:46:32.993 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Updating smithi052:/var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T20:46:32.993 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.993 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.993 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.993 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.993 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.993 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-16T20:46:32.993 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-16T20:46:32.993 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:32.993 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Deploying daemon crash.smithi150 on smithi150 2023-12-16T20:46:32.994 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/3984074714' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:32.994 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.994 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.994 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.994 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.994 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Deploying daemon node-exporter.smithi150 on smithi150 2023-12-16T20:46:32.994 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/2053791766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:32.994 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.994 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.994 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.994 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.994 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.994 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vjiupe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T20:46:32.994 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vjiupe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-12-16T20:46:32.994 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T20:46:32.995 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:32.995 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Deploying daemon mgr.smithi150.vjiupe on smithi150 2023-12-16T20:46:32.995 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.995 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.995 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.995 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/2523335709' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:32.995 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:32.995 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T20:46:32.995 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:32.995 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: Deploying daemon mon.smithi150 on smithi150 2023-12-16T20:46:32.995 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: mon.smithi150@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2023-12-16T20:46:32.995 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:32 smithi150 ceph-mon[88000]: mon.smithi150@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-12-16T20:46:32.996 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:32.992+0000 7fc4eaffd700 1 -- 172.21.15.150:0/1307711935 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4ec01bec0 con 0x7fc4fc0fd9e0 2023-12-16T20:46:33.225 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:33.224+0000 7fc50081d700 1 -- 172.21.15.150:0/1307711935 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4fc106b80 con 0x7fc4fc0fd9e0 2023-12-16T20:46:38.040 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:38.038+0000 7fc4eaffd700 1 -- 172.21.15.150:0/1307711935 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7fc4ec023a00 con 0x7fc4fc0fd9e0 2023-12-16T20:46:38.040 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-12-16T20:46:38.040 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":2,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","modified":"2023-12-16T20:46:32.993129Z","created":"2023-12-16T20:44:38.105612Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi150","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-12-16T20:46:38.043 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:38.041+0000 7fc50081d700 1 -- 172.21.15.150:0/1307711935 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc4e4036820 msgr2=0x7fc4e4038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:38.043 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:38.041+0000 7fc50081d700 1 --2- 172.21.15.150:0/1307711935 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc4e4036820 0x7fc4e4038cd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0009990 tx=0x7fc4f0006e40 comp rx=0 tx=0).stop 2023-12-16T20:46:38.043 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:38.041+0000 7fc50081d700 1 -- 172.21.15.150:0/1307711935 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4fc0fd9e0 msgr2=0x7fc4fc112d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:38.043 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:38.041+0000 7fc50081d700 1 --2- 172.21.15.150:0/1307711935 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4fc0fd9e0 0x7fc4fc112d20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec009300 tx=0x7fc4ec009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:38.043 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:38.041+0000 7fc50081d700 1 -- 172.21.15.150:0/1307711935 shutdown_connections 2023-12-16T20:46:38.044 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:38.041+0000 7fc50081d700 1 --2- 172.21.15.150:0/1307711935 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc4e4036820 0x7fc4e4038cd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:38.044 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:38.041+0000 7fc50081d700 1 --2- 172.21.15.150:0/1307711935 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4fc0fd9e0 0x7fc4fc112d20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:38.044 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:38.041+0000 7fc50081d700 1 -- 172.21.15.150:0/1307711935 >> 172.21.15.150:0/1307711935 conn(0x7fc4fc0f7410 msgr2=0x7fc4fc101200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:38.044 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:38.042+0000 7fc50081d700 1 -- 172.21.15.150:0/1307711935 shutdown_connections 2023-12-16T20:46:38.044 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:46:38.042+0000 7fc50081d700 1 -- 172.21.15.150:0/1307711935 wait complete. 2023-12-16T20:46:38.044 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 2 2023-12-16T20:46:38.296 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2023-12-16T20:46:38.296 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: mon.smithi052 calling monitor election 2023-12-16T20:46:38.296 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:46:38.296 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/1307711935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:38.296 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:46:38.296 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:46:38.296 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: mon.smithi150 calling monitor election 2023-12-16T20:46:38.296 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:46:38.296 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:46:38.296 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.? 172.21.15.150:0/424688912' entity='mgr.smithi150.vjiupe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vjiupe/crt"}]: dispatch 2023-12-16T20:46:38.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:46:38.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: mon.smithi052 is new leader, mons smithi052,smithi150 in quorum (ranks 0,1) 2023-12-16T20:46:38.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: monmap e2: 2 mons at {smithi052=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],smithi150=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T20:46:38.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: fsmap 2023-12-16T20:46:38.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: osdmap e5: 0 total, 0 up, 0 in 2023-12-16T20:46:38.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: mgrmap e17: smithi052.nmkcbs(active, since 20s) 2023-12-16T20:46:38.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: Standby manager daemon smithi150.vjiupe started 2023-12-16T20:46:38.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.? 172.21.15.150:0/424688912' entity='mgr.smithi150.vjiupe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T20:46:38.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: overall HEALTH_OK 2023-12-16T20:46:38.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.? 172.21.15.150:0/424688912' entity='mgr.smithi150.vjiupe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vjiupe/key"}]: dispatch 2023-12-16T20:46:38.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.? 172.21.15.150:0/424688912' entity='mgr.smithi150.vjiupe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T20:46:38.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:38.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:38.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:38.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:38.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:38.298 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-12-16T20:46:38.299 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph config generate-minimal-conf 2023-12-16T20:46:38.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2023-12-16T20:46:38.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: mon.smithi052 calling monitor election 2023-12-16T20:46:38.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:46:38.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/1307711935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T20:46:38.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:46:38.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:46:38.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: mon.smithi150 calling monitor election 2023-12-16T20:46:38.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:46:38.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:46:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.? 172.21.15.150:0/424688912' entity='mgr.smithi150.vjiupe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vjiupe/crt"}]: dispatch 2023-12-16T20:46:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:46:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: mon.smithi052 is new leader, mons smithi052,smithi150 in quorum (ranks 0,1) 2023-12-16T20:46:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: monmap e2: 2 mons at {smithi052=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],smithi150=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T20:46:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: fsmap 2023-12-16T20:46:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: osdmap e5: 0 total, 0 up, 0 in 2023-12-16T20:46:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: mgrmap e17: smithi052.nmkcbs(active, since 20s) 2023-12-16T20:46:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: Standby manager daemon smithi150.vjiupe started 2023-12-16T20:46:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.? 172.21.15.150:0/424688912' entity='mgr.smithi150.vjiupe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T20:46:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: overall HEALTH_OK 2023-12-16T20:46:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.? 172.21.15.150:0/424688912' entity='mgr.smithi150.vjiupe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vjiupe/key"}]: dispatch 2023-12-16T20:46:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.? 172.21.15.150:0/424688912' entity='mgr.smithi150.vjiupe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T20:46:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:38.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:38.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:38.579 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:46:39.364 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:39 smithi052 ceph-mon[80761]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:39.364 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:39 smithi052 ceph-mon[80761]: mgrmap e18: smithi052.nmkcbs(active, since 20s), standbys: smithi150.vjiupe 2023-12-16T20:46:39.364 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:39 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr metadata", "who": "smithi150.vjiupe", "id": "smithi150.vjiupe"}]: dispatch 2023-12-16T20:46:39.364 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:39 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:46:39.364 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:39 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:46:39.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:39 smithi150 ceph-mon[88000]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:39.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:39 smithi150 ceph-mon[88000]: mgrmap e18: smithi052.nmkcbs(active, since 20s), standbys: smithi150.vjiupe 2023-12-16T20:46:39.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:39 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr metadata", "who": "smithi150.vjiupe", "id": "smithi150.vjiupe"}]: dispatch 2023-12-16T20:46:39.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:39 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:46:39.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:39 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:46:39.523 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.521+0000 7f661bbcf700 1 -- 172.21.15.52:0/414523122 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f661406ff80 msgr2=0x7f6614070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:39.523 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.521+0000 7f661bbcf700 1 --2- 172.21.15.52:0/414523122 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f661406ff80 0x7f6614070350 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f66100048c0 tx=0x7f6610009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:39.523 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.521+0000 7f661bbcf700 1 -- 172.21.15.52:0/414523122 shutdown_connections 2023-12-16T20:46:39.523 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.521+0000 7f661bbcf700 1 --2- 172.21.15.52:0/414523122 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f661406ff80 0x7f6614070350 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:39.523 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.521+0000 7f661bbcf700 1 -- 172.21.15.52:0/414523122 >> 172.21.15.52:0/414523122 conn(0x7f66140f8410 msgr2=0x7f66140fa820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:39.523 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.521+0000 7f661bbcf700 1 -- 172.21.15.52:0/414523122 shutdown_connections 2023-12-16T20:46:39.523 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.521+0000 7f661bbcf700 1 -- 172.21.15.52:0/414523122 wait complete. 2023-12-16T20:46:39.524 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.522+0000 7f661bbcf700 1 Processor -- start 2023-12-16T20:46:39.524 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.522+0000 7f661bbcf700 1 -- start start 2023-12-16T20:46:39.524 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.522+0000 7f661bbcf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f661406ff80 0x7f6614114ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:39.524 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.522+0000 7f661bbcf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6614115430 0x7f661410edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:39.525 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.522+0000 7f661996b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f661406ff80 0x7f6614114ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:39.525 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.522+0000 7f661996b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f661406ff80 0x7f6614114ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47928/0 (socket says 172.21.15.52:47928) 2023-12-16T20:46:39.525 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.522+0000 7f661996b700 1 -- 172.21.15.52:0/1336742293 learned_addr learned my addr 172.21.15.52:0/1336742293 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:46:39.525 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.522+0000 7f661bbcf700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6614115a40 con 0x7f661406ff80 2023-12-16T20:46:39.525 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.522+0000 7f661bbcf700 1 -- 172.21.15.52:0/1336742293 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6614115bb0 con 0x7f6614115430 2023-12-16T20:46:39.525 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.523+0000 7f661916a700 1 --2- 172.21.15.52:0/1336742293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6614115430 0x7f661410edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:39.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.523+0000 7f661916a700 1 -- 172.21.15.52:0/1336742293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6614115430 msgr2=0x7f661410edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-12-16T20:46:39.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.523+0000 7f661916a700 1 -- 172.21.15.52:0/1336742293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6614115430 msgr2=0x7f661410edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-12-16T20:46:39.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.523+0000 7f661916a700 1 --2- 172.21.15.52:0/1336742293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6614115430 0x7f661410edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-16T20:46:39.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.523+0000 7f661916a700 1 --2- 172.21.15.52:0/1336742293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6614115430 0x7f661410edd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T20:46:39.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.523+0000 7f661996b700 1 -- 172.21.15.52:0/1336742293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6614115430 msgr2=0x7f661410edd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T20:46:39.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.523+0000 7f661996b700 1 --2- 172.21.15.52:0/1336742293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6614115430 0x7f661410edd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:39.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.523+0000 7f661996b700 1 -- 172.21.15.52:0/1336742293 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6610005040 con 0x7f661406ff80 2023-12-16T20:46:39.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.523+0000 7f661996b700 1 --2- 172.21.15.52:0/1336742293 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f661406ff80 0x7f6614114ef0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f661000bfe0 tx=0x7f6610009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:39.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.523+0000 7f660a7fc700 1 -- 172.21.15.52:0/1336742293 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f661000e040 con 0x7f661406ff80 2023-12-16T20:46:39.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.523+0000 7f660a7fc700 1 -- 172.21.15.52:0/1336742293 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6610013420 con 0x7f661406ff80 2023-12-16T20:46:39.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.523+0000 7f661bbcf700 1 -- 172.21.15.52:0/1336742293 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661410f3d0 con 0x7f661406ff80 2023-12-16T20:46:39.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.523+0000 7f660a7fc700 1 -- 172.21.15.52:0/1336742293 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f661001c630 con 0x7f661406ff80 2023-12-16T20:46:39.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.524+0000 7f661bbcf700 1 -- 172.21.15.52:0/1336742293 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661410f910 con 0x7f661406ff80 2023-12-16T20:46:39.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.524+0000 7f661bbcf700 1 -- 172.21.15.52:0/1336742293 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66141063f0 con 0x7f661406ff80 2023-12-16T20:46:39.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.525+0000 7f660a7fc700 1 -- 172.21.15.52:0/1336742293 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f66100139e0 con 0x7f661406ff80 2023-12-16T20:46:39.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.526+0000 7f660a7fc700 1 --2- 172.21.15.52:0/1336742293 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6600069030 0x7f660006b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:39.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.526+0000 7f660a7fc700 1 -- 172.21.15.52:0/1336742293 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6610086f80 con 0x7f661406ff80 2023-12-16T20:46:39.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.526+0000 7f661916a700 1 --2- 172.21.15.52:0/1336742293 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6600069030 0x7f660006b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:39.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.527+0000 7f661916a700 1 --2- 172.21.15.52:0/1336742293 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6600069030 0x7f660006b4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6604006f30 tx=0x7f6604008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:39.533 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.531+0000 7f660a7fc700 1 -- 172.21.15.52:0/1336742293 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f66100b8a00 con 0x7f661406ff80 2023-12-16T20:46:39.734 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.731+0000 7f661bbcf700 1 -- 172.21.15.52:0/1336742293 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f661410fb80 con 0x7f661406ff80 2023-12-16T20:46:39.734 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.732+0000 7f660a7fc700 1 -- 172.21.15.52:0/1336742293 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7f66100545d0 con 0x7f661406ff80 2023-12-16T20:46:39.734 INFO:teuthology.orchestra.run.smithi052.stdout:# minimal ceph.conf for cf043a8a-9c53-11ee-95a5-87774f69a715 2023-12-16T20:46:39.734 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2023-12-16T20:46:39.734 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = cf043a8a-9c53-11ee-95a5-87774f69a715 2023-12-16T20:46:39.735 INFO:teuthology.orchestra.run.smithi052.stdout: mon_host = [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-12-16T20:46:39.736 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.734+0000 7f661bbcf700 1 -- 172.21.15.52:0/1336742293 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6600069030 msgr2=0x7f660006b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:39.736 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.734+0000 7f661bbcf700 1 --2- 172.21.15.52:0/1336742293 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6600069030 0x7f660006b4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6604006f30 tx=0x7f6604008040 comp rx=0 tx=0).stop 2023-12-16T20:46:39.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.734+0000 7f661bbcf700 1 -- 172.21.15.52:0/1336742293 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f661406ff80 msgr2=0x7f6614114ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:39.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.734+0000 7f661bbcf700 1 --2- 172.21.15.52:0/1336742293 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f661406ff80 0x7f6614114ef0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f661000bfe0 tx=0x7f6610009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:39.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.734+0000 7f661bbcf700 1 -- 172.21.15.52:0/1336742293 shutdown_connections 2023-12-16T20:46:39.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.734+0000 7f661bbcf700 1 --2- 172.21.15.52:0/1336742293 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6600069030 0x7f660006b4e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:39.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.734+0000 7f661bbcf700 1 --2- 172.21.15.52:0/1336742293 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f661406ff80 0x7f6614114ef0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:39.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.734+0000 7f661bbcf700 1 --2- 172.21.15.52:0/1336742293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6614115430 0x7f661410edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:39.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.734+0000 7f661bbcf700 1 -- 172.21.15.52:0/1336742293 >> 172.21.15.52:0/1336742293 conn(0x7f66140f8410 msgr2=0x7f6614103570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:39.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.734+0000 7f661bbcf700 1 -- 172.21.15.52:0/1336742293 shutdown_connections 2023-12-16T20:46:39.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:39.734+0000 7f661bbcf700 1 -- 172.21.15.52:0/1336742293 wait complete. 2023-12-16T20:46:40.067 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-12-16T20:46:40.067 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:46:40.067 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.conf 2023-12-16T20:46:40.101 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:46:40.101 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-16T20:46:40.173 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-16T20:46:40.173 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.conf 2023-12-16T20:46:40.203 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-16T20:46:40.203 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-16T20:46:40.272 INFO:tasks.cephadm:Deploying OSDs... 2023-12-16T20:46:40.272 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:46:40.272 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2023-12-16T20:46:40.288 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-16T20:46:40.289 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 2023-12-16T20:46:40.345 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-16T20:46:40.345 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T20:46:40.345 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-12-16T20:46:40.345 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T20:46:40.345 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T20:46:40.346 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2023-12-16 20:46:09.516169326 +0000 2023-12-16T20:46:40.346 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2023-12-16 20:41:04.711381879 +0000 2023-12-16T20:46:40.346 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2023-12-16 20:41:04.711381879 +0000 2023-12-16T20:46:40.346 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2023-12-16 20:41:04.711381879 +0000 2023-12-16T20:46:40.346 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-16T20:46:40.415 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2023-12-16T20:46:40.415 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2023-12-16T20:46:40.415 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 8.6917e-05 s, 5.9 MB/s 2023-12-16T20:46:40.416 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-16T20:46:40.472 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 2023-12-16T20:46:40.530 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-16T20:46:40.530 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T20:46:40.530 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-12-16T20:46:40.530 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T20:46:40.530 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T20:46:40.531 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2023-12-16 20:46:09.516169326 +0000 2023-12-16T20:46:40.531 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2023-12-16 20:41:05.214375857 +0000 2023-12-16T20:46:40.531 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2023-12-16 20:41:05.214375857 +0000 2023-12-16T20:46:40.531 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2023-12-16 20:41:05.214375857 +0000 2023-12-16T20:46:40.531 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-16T20:46:40.601 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2023-12-16T20:46:40.601 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2023-12-16T20:46:40.601 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000101026 s, 5.1 MB/s 2023-12-16T20:46:40.602 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-16T20:46:40.659 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 2023-12-16T20:46:40.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:40.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:40.689 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:40.689 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:46:40.689 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: Updating smithi052:/etc/ceph/ceph.conf 2023-12-16T20:46:40.689 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: Updating smithi150:/etc/ceph/ceph.conf 2023-12-16T20:46:40.689 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: Updating smithi150:/var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T20:46:40.689 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: Updating smithi052:/var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T20:46:40.689 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/1336742293' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:40.689 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:40.689 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:40.689 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:40.689 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:40.689 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:40.689 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:40.690 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: Reconfiguring mon.smithi052 (unknown last config time)... 2023-12-16T20:46:40.690 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T20:46:40.690 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T20:46:40.690 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:40.690 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:40 smithi150 ceph-mon[88000]: Reconfiguring daemon mon.smithi052 on smithi052 2023-12-16T20:46:40.714 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-16T20:46:40.714 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T20:46:40.714 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 702 Links: 1 2023-12-16T20:46:40.714 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T20:46:40.714 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T20:46:40.715 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2023-12-16 20:46:09.517169314 +0000 2023-12-16T20:46:40.715 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2023-12-16 20:41:05.714369870 +0000 2023-12-16T20:46:40.715 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2023-12-16 20:41:05.714369870 +0000 2023-12-16T20:46:40.715 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2023-12-16 20:41:05.714369870 +0000 2023-12-16T20:46:40.715 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-16T20:46:40.751 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:40.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:40.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:40.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:46:40.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: Updating smithi052:/etc/ceph/ceph.conf 2023-12-16T20:46:40.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: Updating smithi150:/etc/ceph/ceph.conf 2023-12-16T20:46:40.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: Updating smithi150:/var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T20:46:40.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: Updating smithi052:/var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T20:46:40.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/1336742293' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:40.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:40.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:40.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:40.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:40.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:40.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:40.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: Reconfiguring mon.smithi052 (unknown last config time)... 2023-12-16T20:46:40.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T20:46:40.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T20:46:40.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:40.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:40 smithi052 ceph-mon[80761]: Reconfiguring daemon mon.smithi052 on smithi052 2023-12-16T20:46:40.781 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2023-12-16T20:46:40.781 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2023-12-16T20:46:40.781 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 8.6857e-05 s, 5.9 MB/s 2023-12-16T20:46:40.782 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-16T20:46:40.839 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 2023-12-16T20:46:40.894 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-16T20:46:40.894 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T20:46:40.894 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-12-16T20:46:40.894 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T20:46:40.895 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T20:46:40.895 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2023-12-16 20:46:09.517169314 +0000 2023-12-16T20:46:40.895 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2023-12-16 20:41:06.213363896 +0000 2023-12-16T20:46:40.895 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2023-12-16 20:41:06.213363896 +0000 2023-12-16T20:46:40.895 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2023-12-16 20:41:06.213363896 +0000 2023-12-16T20:46:40.895 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-16T20:46:40.965 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2023-12-16T20:46:40.965 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2023-12-16T20:46:40.965 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.0591e-05 s, 5.7 MB/s 2023-12-16T20:46:40.966 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-16T20:46:41.022 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-16T20:46:41.022 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2023-12-16T20:46:41.042 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-16T20:46:41.042 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2023-12-16T20:46:41.100 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-16T20:46:41.100 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T20:46:41.100 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-12-16T20:46:41.100 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T20:46:41.100 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T20:46:41.100 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-12-16 20:46:32.170819458 +0000 2023-12-16T20:46:41.101 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-12-16 20:40:57.096395460 +0000 2023-12-16T20:46:41.101 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-12-16 20:40:57.096395460 +0000 2023-12-16T20:46:41.101 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2023-12-16 20:40:57.096395460 +0000 2023-12-16T20:46:41.101 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-16T20:46:41.170 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-12-16T20:46:41.171 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-12-16T20:46:41.171 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000240637 s, 2.1 MB/s 2023-12-16T20:46:41.172 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-16T20:46:41.233 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2023-12-16T20:46:41.291 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-16T20:46:41.292 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T20:46:41.292 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-12-16T20:46:41.292 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T20:46:41.292 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T20:46:41.292 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-12-16 20:46:32.170819458 +0000 2023-12-16T20:46:41.292 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-12-16 20:40:57.590385556 +0000 2023-12-16T20:46:41.292 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-12-16 20:40:57.590385556 +0000 2023-12-16T20:46:41.292 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2023-12-16 20:40:57.590385556 +0000 2023-12-16T20:46:41.292 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-16T20:46:41.360 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-12-16T20:46:41.360 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-12-16T20:46:41.360 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000215905 s, 2.4 MB/s 2023-12-16T20:46:41.362 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-16T20:46:41.420 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2023-12-16T20:46:41.477 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-16T20:46:41.477 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T20:46:41.477 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 702 Links: 1 2023-12-16T20:46:41.477 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T20:46:41.477 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T20:46:41.477 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-12-16 20:46:32.170819458 +0000 2023-12-16T20:46:41.477 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-12-16 20:40:58.078375771 +0000 2023-12-16T20:46:41.477 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-12-16 20:40:58.078375771 +0000 2023-12-16T20:46:41.477 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2023-12-16 20:40:58.078375771 +0000 2023-12-16T20:46:41.478 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-16T20:46:41.546 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-12-16T20:46:41.546 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-12-16T20:46:41.546 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000212855 s, 2.4 MB/s 2023-12-16T20:46:41.547 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-16T20:46:41.605 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2023-12-16T20:46:41.662 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-16T20:46:41.662 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T20:46:41.662 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-12-16T20:46:41.662 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T20:46:41.662 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T20:46:41.662 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-12-16 20:46:32.171819439 +0000 2023-12-16T20:46:41.662 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-12-16 20:40:58.562366067 +0000 2023-12-16T20:46:41.663 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-12-16 20:40:58.562366067 +0000 2023-12-16T20:46:41.663 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2023-12-16 20:40:58.562366067 +0000 2023-12-16T20:46:41.663 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-16T20:46:41.730 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-12-16T20:46:41.731 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-12-16T20:46:41.731 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000213758 s, 2.4 MB/s 2023-12-16T20:46:41.732 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-16T20:46:41.789 INFO:tasks.cephadm:Deploying osd.0 on smithi052 with /dev/vg_nvme/lv_4... 2023-12-16T20:46:41.789 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-12-16T20:46:42.054 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:46:42.217 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:41 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:42.217 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:41 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:42.217 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:41 smithi052 ceph-mon[80761]: Reconfiguring mgr.smithi052.nmkcbs (unknown last config time)... 2023-12-16T20:46:42.217 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:41 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi052.nmkcbs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T20:46:42.217 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:41 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T20:46:42.217 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:41 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:42.217 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:41 smithi052 ceph-mon[80761]: Reconfiguring daemon mgr.smithi052.nmkcbs on smithi052 2023-12-16T20:46:42.217 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:41 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:42.217 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:41 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:42.217 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:41 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi052", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-16T20:46:42.217 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:41 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:42.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:41 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:42.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:41 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:42.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:41 smithi150 ceph-mon[88000]: Reconfiguring mgr.smithi052.nmkcbs (unknown last config time)... 2023-12-16T20:46:42.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:41 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi052.nmkcbs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T20:46:42.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:41 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T20:46:42.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:41 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:42.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:41 smithi150 ceph-mon[88000]: Reconfiguring daemon mgr.smithi052.nmkcbs on smithi052 2023-12-16T20:46:42.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:41 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:42.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:41 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:42.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:41 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi052", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-16T20:46:42.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:41 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:43.191 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:42 smithi052 ceph-mon[80761]: Reconfiguring crash.smithi052 (monmap changed)... 2023-12-16T20:46:43.191 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:42 smithi052 ceph-mon[80761]: Reconfiguring daemon crash.smithi052 on smithi052 2023-12-16T20:46:43.191 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:42 smithi052 ceph-mon[80761]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:43.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:42 smithi150 ceph-mon[88000]: Reconfiguring crash.smithi052 (monmap changed)... 2023-12-16T20:46:43.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:42 smithi150 ceph-mon[88000]: Reconfiguring daemon crash.smithi052 on smithi052 2023-12-16T20:46:43.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:42 smithi150 ceph-mon[88000]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:43.724 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-12-16T20:46:43.724 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-12-16T20:46:43.725 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2023-12-16T20:46:43.725 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2023-12-16T20:46:43.725 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00815527 s, 1.3 GB/s 2023-12-16T20:46:43.725 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2023-12-16T20:46:44.091 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:43 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:44.091 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:43 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:44.091 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:43 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:44.236 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:43 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:44.236 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:43 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:44.236 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:43 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:44.875 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi052:vg_nvme/lv_4 2023-12-16T20:46:45.198 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:46:45.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:44 smithi052 ceph-mon[80761]: Reconfiguring alertmanager.smithi052 (dependencies changed)... 2023-12-16T20:46:45.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:44 smithi052 ceph-mon[80761]: Reconfiguring daemon alertmanager.smithi052 on smithi052 2023-12-16T20:46:45.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:44 smithi052 ceph-mon[80761]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:45.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:44 smithi150 ceph-mon[88000]: Reconfiguring alertmanager.smithi052 (dependencies changed)... 2023-12-16T20:46:45.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:44 smithi150 ceph-mon[88000]: Reconfiguring daemon alertmanager.smithi052 on smithi052 2023-12-16T20:46:45.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:44 smithi150 ceph-mon[88000]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:46.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.082+0000 7f484bb6d700 1 -- 172.21.15.52:0/3134335617 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48440fcbf0 msgr2=0x7f48440fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:46.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.082+0000 7f484bb6d700 1 --2- 172.21.15.52:0/3134335617 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48440fcbf0 0x7f48440fd000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4834004940 tx=0x7f4834009e30 comp rx=0 tx=0).stop 2023-12-16T20:46:46.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.082+0000 7f484bb6d700 1 -- 172.21.15.52:0/3134335617 shutdown_connections 2023-12-16T20:46:46.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.082+0000 7f484bb6d700 1 --2- 172.21.15.52:0/3134335617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48440fde00 0x7f48440fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:46.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.082+0000 7f484bb6d700 1 --2- 172.21.15.52:0/3134335617 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48440fcbf0 0x7f48440fd000 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:46.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.082+0000 7f484bb6d700 1 -- 172.21.15.52:0/3134335617 >> 172.21.15.52:0/3134335617 conn(0x7f48440f82d0 msgr2=0x7f48440fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:46:46.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.082+0000 7f484bb6d700 1 -- 172.21.15.52:0/3134335617 shutdown_connections 2023-12-16T20:46:46.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.082+0000 7f484bb6d700 1 -- 172.21.15.52:0/3134335617 wait complete. 2023-12-16T20:46:46.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.082+0000 7f484bb6d700 1 Processor -- start 2023-12-16T20:46:46.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.082+0000 7f484bb6d700 1 -- start start 2023-12-16T20:46:46.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f484bb6d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48440fcbf0 0x7f484410d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:46.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f484bb6d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48440fde00 0x7f484410daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:46.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f484bb6d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484410e0a0 con 0x7f48440fcbf0 2023-12-16T20:46:46.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f484bb6d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484410e940 con 0x7f48440fde00 2023-12-16T20:46:46.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f4849909700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48440fcbf0 0x7f484410d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:46.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f4849909700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48440fcbf0 0x7f484410d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42650/0 (socket says 172.21.15.52:42650) 2023-12-16T20:46:46.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f4849909700 1 -- 172.21.15.52:0/1515716923 learned_addr learned my addr 172.21.15.52:0/1515716923 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:46:46.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f4849108700 1 --2- 172.21.15.52:0/1515716923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48440fde00 0x7f484410daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:46.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f4849909700 1 -- 172.21.15.52:0/1515716923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48440fde00 msgr2=0x7f484410daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:46:46.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f4849909700 1 --2- 172.21.15.52:0/1515716923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48440fde00 0x7f484410daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:46:46.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f4849909700 1 -- 172.21.15.52:0/1515716923 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4834005040 con 0x7f48440fcbf0 2023-12-16T20:46:46.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f4849909700 1 --2- 172.21.15.52:0/1515716923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48440fcbf0 0x7f484410d560 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4834009360 tx=0x7f4834009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:46.087 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f483a7fc700 1 -- 172.21.15.52:0/1515716923 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f483400e040 con 0x7f48440fcbf0 2023-12-16T20:46:46.087 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f483a7fc700 1 -- 172.21.15.52:0/1515716923 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4834004290 con 0x7f48440fcbf0 2023-12-16T20:46:46.087 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f483a7fc700 1 -- 172.21.15.52:0/1515716923 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f483401c630 con 0x7f48440fcbf0 2023-12-16T20:46:46.087 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.083+0000 7f484bb6d700 1 -- 172.21.15.52:0/1515716923 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f484410ebc0 con 0x7f48440fcbf0 2023-12-16T20:46:46.087 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.084+0000 7f484bb6d700 1 -- 172.21.15.52:0/1515716923 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484410f100 con 0x7f48440fcbf0 2023-12-16T20:46:46.088 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.085+0000 7f484bb6d700 1 -- 172.21.15.52:0/1515716923 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48441075d0 con 0x7f48440fcbf0 2023-12-16T20:46:46.088 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.085+0000 7f483a7fc700 1 -- 172.21.15.52:0/1515716923 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f4834013420 con 0x7f48440fcbf0 2023-12-16T20:46:46.088 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.085+0000 7f483a7fc700 1 --2- 172.21.15.52:0/1515716923 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4830068ed0 0x7f483006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:46:46.088 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.085+0000 7f483a7fc700 1 -- 172.21.15.52:0/1515716923 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f4834086f60 con 0x7f48440fcbf0 2023-12-16T20:46:46.088 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.086+0000 7f4849108700 1 --2- 172.21.15.52:0/1515716923 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4830068ed0 0x7f483006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:46:46.088 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.086+0000 7f4849108700 1 --2- 172.21.15.52:0/1515716923 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4830068ed0 0x7f483006b380 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4840007fd0 tx=0x7f4840009040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:46:46.091 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.090+0000 7f483a7fc700 1 -- 172.21.15.52:0/1515716923 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4834058040 con 0x7f48440fcbf0 2023-12-16T20:46:46.306 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:46:46.305+0000 7f484bb6d700 1 -- 172.21.15.52:0/1515716923 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f484410f370 con 0x7f4830068ed0 2023-12-16T20:46:47.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:46 smithi052 ceph-mon[80761]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:47.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T20:46:47.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T20:46:47.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:47.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:46 smithi150 ceph-mon[88000]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:47.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T20:46:47.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T20:46:47.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:48.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:47 smithi052 ceph-mon[80761]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:46:48.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:47 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:48.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:47 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:48.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:47 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:46:48.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:47 smithi150 ceph-mon[88000]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:46:48.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:47 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:48.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:47 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:48.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:47 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:46:49.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:48 smithi052 ceph-mon[80761]: Reconfiguring grafana.smithi052 (dependencies changed)... 2023-12-16T20:46:49.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:48 smithi052 ceph-mon[80761]: Reconfiguring daemon grafana.smithi052 on smithi052 2023-12-16T20:46:49.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:48 smithi052 ceph-mon[80761]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:49.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:48 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/4057202962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03268674-f4fa-4c69-9fa5-ac2327873b9e"}]: dispatch 2023-12-16T20:46:49.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:48 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/4057202962' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03268674-f4fa-4c69-9fa5-ac2327873b9e"}]': finished 2023-12-16T20:46:49.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:48 smithi052 ceph-mon[80761]: osdmap e6: 1 total, 0 up, 1 in 2023-12-16T20:46:49.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:48 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T20:46:49.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:48 smithi150 ceph-mon[88000]: Reconfiguring grafana.smithi052 (dependencies changed)... 2023-12-16T20:46:49.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:48 smithi150 ceph-mon[88000]: Reconfiguring daemon grafana.smithi052 on smithi052 2023-12-16T20:46:49.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:48 smithi150 ceph-mon[88000]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:49.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:48 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/4057202962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03268674-f4fa-4c69-9fa5-ac2327873b9e"}]: dispatch 2023-12-16T20:46:49.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:48 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/4057202962' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03268674-f4fa-4c69-9fa5-ac2327873b9e"}]': finished 2023-12-16T20:46:49.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:48 smithi150 ceph-mon[88000]: osdmap e6: 1 total, 0 up, 1 in 2023-12-16T20:46:49.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:48 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T20:46:50.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:49 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3650520692' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T20:46:50.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:49 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/3650520692' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T20:46:51.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:50 smithi052 ceph-mon[80761]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:51.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:50 smithi150 ceph-mon[88000]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:53.132 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:52 smithi052 ceph-mon[80761]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:53.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:52 smithi150 ceph-mon[88000]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:55.226 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:54 smithi052 ceph-mon[80761]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:55.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:54 smithi150 ceph-mon[88000]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:56.220 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:56 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:56.220 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:56 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:56.220 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:56 smithi052 ceph-mon[80761]: Reconfiguring prometheus.smithi052 (dependencies changed)... 2023-12-16T20:46:56.220 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:56 smithi052 ceph-mon[80761]: Reconfiguring daemon prometheus.smithi052 on smithi052 2023-12-16T20:46:56.221 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:56 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-16T20:46:56.221 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:56 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:56.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:56 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:56.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:56 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:56.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:56 smithi150 ceph-mon[88000]: Reconfiguring prometheus.smithi052 (dependencies changed)... 2023-12-16T20:46:56.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:56 smithi150 ceph-mon[88000]: Reconfiguring daemon prometheus.smithi052 on smithi052 2023-12-16T20:46:56.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:56 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-16T20:46:56.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:56 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:57.106 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:57 smithi052 ceph-mon[80761]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:57.106 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:57 smithi052 ceph-mon[80761]: Deploying daemon osd.0 on smithi052 2023-12-16T20:46:57.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:57 smithi150 ceph-mon[88000]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:57.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:57 smithi150 ceph-mon[88000]: Deploying daemon osd.0 on smithi052 2023-12-16T20:46:58.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:58 smithi150 ceph-mon[88000]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:58.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:58 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:58.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:58 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:58.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:58 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-16T20:46:58.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:58 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:46:58.731 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:58 smithi052 ceph-mon[80761]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:46:58.731 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:58 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:58.731 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:58 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:46:58.731 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:58 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-16T20:46:58.731 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:58 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:00.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:59 smithi150 ceph-mon[88000]: Reconfiguring crash.smithi150 (monmap changed)... 2023-12-16T20:47:00.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:59 smithi150 ceph-mon[88000]: Reconfiguring daemon crash.smithi150 on smithi150 2023-12-16T20:47:00.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:00.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:00.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:59 smithi150 ceph-mon[88000]: Reconfiguring mgr.smithi150.vjiupe (monmap changed)... 2023-12-16T20:47:00.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vjiupe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T20:47:00.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T20:47:00.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:00.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:59 smithi150 ceph-mon[88000]: Reconfiguring daemon mgr.smithi150.vjiupe on smithi150 2023-12-16T20:47:00.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:00.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:00.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T20:47:00.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T20:47:00.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:46:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:59 smithi052 ceph-mon[80761]: Reconfiguring crash.smithi150 (monmap changed)... 2023-12-16T20:47:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:59 smithi052 ceph-mon[80761]: Reconfiguring daemon crash.smithi150 on smithi150 2023-12-16T20:47:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:59 smithi052 ceph-mon[80761]: Reconfiguring mgr.smithi150.vjiupe (monmap changed)... 2023-12-16T20:47:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vjiupe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T20:47:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T20:47:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:59 smithi052 ceph-mon[80761]: Reconfiguring daemon mgr.smithi150.vjiupe on smithi150 2023-12-16T20:47:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:00.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T20:47:00.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T20:47:00.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:46:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:01.134 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:00 smithi150 ceph-mon[88000]: Reconfiguring mon.smithi150 (monmap changed)... 2023-12-16T20:47:01.134 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:00 smithi150 ceph-mon[88000]: Reconfiguring daemon mon.smithi150 on smithi150 2023-12-16T20:47:01.135 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:00 smithi150 ceph-mon[88000]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:47:01.135 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:01.135 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:01.135 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T20:47:01.135 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi052.front.sepia.ceph.com:9093"}]: dispatch 2023-12-16T20:47:01.135 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:01.135 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T20:47:01.135 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi052.front.sepia.ceph.com:3000"}]: dispatch 2023-12-16T20:47:01.135 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:01.135 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T20:47:01.135 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi052.front.sepia.ceph.com:9095"}]: dispatch 2023-12-16T20:47:01.135 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:01.135 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:47:01.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:00 smithi052 ceph-mon[80761]: Reconfiguring mon.smithi150 (monmap changed)... 2023-12-16T20:47:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:00 smithi052 ceph-mon[80761]: Reconfiguring daemon mon.smithi150 on smithi150 2023-12-16T20:47:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:00 smithi052 ceph-mon[80761]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:47:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T20:47:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi052.front.sepia.ceph.com:9093"}]: dispatch 2023-12-16T20:47:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T20:47:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi052.front.sepia.ceph.com:3000"}]: dispatch 2023-12-16T20:47:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T20:47:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi052.front.sepia.ceph.com:9095"}]: dispatch 2023-12-16T20:47:01.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:01.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:47:02.178 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:01 smithi052 ceph-mon[80761]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T20:47:02.178 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:01 smithi052 ceph-mon[80761]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi052.front.sepia.ceph.com:9093"}]: dispatch 2023-12-16T20:47:02.178 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:01 smithi052 ceph-mon[80761]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T20:47:02.179 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:01 smithi052 ceph-mon[80761]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi052.front.sepia.ceph.com:3000"}]: dispatch 2023-12-16T20:47:02.179 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:01 smithi052 ceph-mon[80761]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T20:47:02.179 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:01 smithi052 ceph-mon[80761]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi052.front.sepia.ceph.com:9095"}]: dispatch 2023-12-16T20:47:02.179 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:01 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:02.179 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:01 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:02.179 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:01 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:02.179 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:01 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:02.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:01 smithi150 ceph-mon[88000]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T20:47:02.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:01 smithi150 ceph-mon[88000]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi052.front.sepia.ceph.com:9093"}]: dispatch 2023-12-16T20:47:02.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:01 smithi150 ceph-mon[88000]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T20:47:02.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:01 smithi150 ceph-mon[88000]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi052.front.sepia.ceph.com:3000"}]: dispatch 2023-12-16T20:47:02.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:01 smithi150 ceph-mon[88000]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T20:47:02.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:01 smithi150 ceph-mon[88000]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi052.front.sepia.ceph.com:9095"}]: dispatch 2023-12-16T20:47:02.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:01 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:02.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:01 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:02.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:01 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:02.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:01 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:03.029 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:02 smithi052 ceph-mon[80761]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:47:03.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:02 smithi150 ceph-mon[88000]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:47:03.978 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:03.977+0000 7f483a7fc700 1 -- 172.21.15.52:0/1515716923 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f484410f370 con 0x7f4830068ed0 2023-12-16T20:47:03.978 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 0 on host 'smithi052' 2023-12-16T20:47:03.980 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:03.979+0000 7f484bb6d700 1 -- 172.21.15.52:0/1515716923 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4830068ed0 msgr2=0x7f483006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:47:03.980 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:03.979+0000 7f484bb6d700 1 --2- 172.21.15.52:0/1515716923 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4830068ed0 0x7f483006b380 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4840007fd0 tx=0x7f4840009040 comp rx=0 tx=0).stop 2023-12-16T20:47:03.980 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:03.979+0000 7f484bb6d700 1 -- 172.21.15.52:0/1515716923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48440fcbf0 msgr2=0x7f484410d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:47:03.980 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:03.979+0000 7f484bb6d700 1 --2- 172.21.15.52:0/1515716923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48440fcbf0 0x7f484410d560 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4834009360 tx=0x7f4834009e30 comp rx=0 tx=0).stop 2023-12-16T20:47:03.981 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:03.979+0000 7f484bb6d700 1 -- 172.21.15.52:0/1515716923 shutdown_connections 2023-12-16T20:47:03.981 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:03.979+0000 7f484bb6d700 1 --2- 172.21.15.52:0/1515716923 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4830068ed0 0x7f483006b380 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:03.981 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:03.979+0000 7f484bb6d700 1 --2- 172.21.15.52:0/1515716923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48440fcbf0 0x7f484410d560 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:03.981 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:03.979+0000 7f484bb6d700 1 --2- 172.21.15.52:0/1515716923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48440fde00 0x7f484410daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:03.981 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:03.979+0000 7f484bb6d700 1 -- 172.21.15.52:0/1515716923 >> 172.21.15.52:0/1515716923 conn(0x7f48440f82d0 msgr2=0x7f48440fa590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:47:03.981 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:03.979+0000 7f484bb6d700 1 -- 172.21.15.52:0/1515716923 shutdown_connections 2023-12-16T20:47:03.981 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:03.979+0000 7f484bb6d700 1 -- 172.21.15.52:0/1515716923 wait complete. 2023-12-16T20:47:04.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:03 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:47:04.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:03 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:47:04.421 DEBUG:teuthology.orchestra.run.smithi052:osd.0> sudo journalctl -f -n 0 -u ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@osd.0.service 2023-12-16T20:47:04.424 INFO:tasks.cephadm:Deploying osd.1 on smithi052 with /dev/vg_nvme/lv_3... 2023-12-16T20:47:04.424 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-12-16T20:47:04.757 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:47:05.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:04 smithi052 ceph-mon[80761]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:47:05.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:04 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:05.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:04 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:05.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:04 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:05.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:04 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:05.253 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:04 smithi150 ceph-mon[88000]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:47:05.253 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:04 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:05.254 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:04 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:05.254 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:04 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:05.254 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:04 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:05.809 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:47:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:47:05.665+0000 7f9c6669b540 -1 osd.0 0 log_to_monitors true 2023-12-16T20:47:06.163 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:05 smithi052 ceph-mon[80761]: from='osd.0 [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-16T20:47:06.255 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:05 smithi150 ceph-mon[88000]: from='osd.0 [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-16T20:47:06.276 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-12-16T20:47:06.276 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-12-16T20:47:06.277 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2023-12-16T20:47:06.277 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2023-12-16T20:47:06.277 INFO:teuthology.orchestra.run.smithi052.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0100558 s, 1.0 GB/s 2023-12-16T20:47:06.277 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 2023-12-16T20:47:06.277 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2023-12-16T20:47:06.747 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi052:vg_nvme/lv_3 2023-12-16T20:47:07.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:47:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:47:06.956+0000 7f9c5e683700 -1 osd.0 0 waiting for initial osdmap 2023-12-16T20:47:07.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:47:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:47:06.960+0000 7f9c56c79700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T20:47:07.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:06 smithi052 ceph-mon[80761]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:47:07.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:06 smithi052 ceph-mon[80761]: from='osd.0 [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-16T20:47:07.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:06 smithi052 ceph-mon[80761]: osdmap e7: 1 total, 0 up, 1 in 2023-12-16T20:47:07.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:06 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T20:47:07.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:06 smithi052 ceph-mon[80761]: from='osd.0 [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2023-12-16T20:47:07.021 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:47:07.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:06 smithi150 ceph-mon[88000]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:47:07.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:06 smithi150 ceph-mon[88000]: from='osd.0 [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-16T20:47:07.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:06 smithi150 ceph-mon[88000]: osdmap e7: 1 total, 0 up, 1 in 2023-12-16T20:47:07.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:06 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T20:47:07.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:06 smithi150 ceph-mon[88000]: from='osd.0 [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2023-12-16T20:47:08.224 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.222+0000 7f93a3b5c700 1 -- 172.21.15.52:0/3342849928 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f939c0fbaf0 msgr2=0x7f939c0fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:47:08.224 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.222+0000 7f93a3b5c700 1 --2- 172.21.15.52:0/3342849928 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f939c0fbaf0 0x7f939c0fbf00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9398004940 tx=0x7f9398009e30 comp rx=0 tx=0).stop 2023-12-16T20:47:08.224 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.222+0000 7f93a3b5c700 1 -- 172.21.15.52:0/3342849928 shutdown_connections 2023-12-16T20:47:08.224 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.222+0000 7f93a3b5c700 1 --2- 172.21.15.52:0/3342849928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f939c0fcd00 0x7f939c0fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:08.224 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.222+0000 7f93a3b5c700 1 --2- 172.21.15.52:0/3342849928 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f939c0fbaf0 0x7f939c0fbf00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:08.224 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.222+0000 7f93a3b5c700 1 -- 172.21.15.52:0/3342849928 >> 172.21.15.52:0/3342849928 conn(0x7f939c0f71d0 msgr2=0x7f939c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:47:08.225 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.222+0000 7f93a3b5c700 1 -- 172.21.15.52:0/3342849928 shutdown_connections 2023-12-16T20:47:08.225 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.222+0000 7f93a3b5c700 1 -- 172.21.15.52:0/3342849928 wait complete. 2023-12-16T20:47:08.225 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:07 smithi052 ceph-mon[80761]: from='osd.0 [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2023-12-16T20:47:08.225 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:07 smithi052 ceph-mon[80761]: osdmap e8: 1 total, 0 up, 1 in 2023-12-16T20:47:08.225 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:07 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T20:47:08.225 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:07 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T20:47:08.225 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:07 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:08.225 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:07 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:08.226 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:07 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:47:08.226 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:07 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:08.226 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:07 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:08.226 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:07 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:47:08.226 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:07 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:08.226 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:07 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:47:08.227 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.223+0000 7f93a3b5c700 1 Processor -- start 2023-12-16T20:47:08.227 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.223+0000 7f93a3b5c700 1 -- start start 2023-12-16T20:47:08.227 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.223+0000 7f93a3b5c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f939c0fbaf0 0x7f939c10c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:47:08.227 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.223+0000 7f93a3b5c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f939c0fcd00 0x7f939c10c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:47:08.228 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.224+0000 7f93a10f7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f939c0fcd00 0x7f939c10c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:47:08.228 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.224+0000 7f93a10f7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f939c0fcd00 0x7f939c10c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37034/0 (socket says 172.21.15.52:37034) 2023-12-16T20:47:08.228 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.224+0000 7f93a18f8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f939c0fbaf0 0x7f939c10c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:47:08.228 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.224+0000 7f93a18f8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f939c0fbaf0 0x7f939c10c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.52:43656/0 (socket says 172.21.15.52:43656) 2023-12-16T20:47:08.228 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.224+0000 7f93a10f7700 1 -- 172.21.15.52:0/1876175595 learned_addr learned my addr 172.21.15.52:0/1876175595 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:47:08.228 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.224+0000 7f93a3b5c700 1 -- 172.21.15.52:0/1876175595 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c10cf10 con 0x7f939c0fcd00 2023-12-16T20:47:08.228 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.224+0000 7f93a3b5c700 1 -- 172.21.15.52:0/1876175595 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c10d7b0 con 0x7f939c0fbaf0 2023-12-16T20:47:08.228 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.224+0000 7f93a18f8700 1 -- 172.21.15.52:0/1876175595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f939c0fcd00 msgr2=0x7f939c10c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:47:08.228 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.224+0000 7f93a18f8700 1 --2- 172.21.15.52:0/1876175595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f939c0fcd00 0x7f939c10c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:08.228 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.224+0000 7f93a18f8700 1 -- 172.21.15.52:0/1876175595 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9398005040 con 0x7f939c0fbaf0 2023-12-16T20:47:08.229 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.224+0000 7f93a10f7700 1 --2- 172.21.15.52:0/1876175595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f939c0fcd00 0x7f939c10c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T20:47:08.229 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.224+0000 7f93a18f8700 1 --2- 172.21.15.52:0/1876175595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f939c0fbaf0 0x7f939c10c3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9398000c00 tx=0x7f9398009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:47:08.229 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.225+0000 7f93927fc700 1 -- 172.21.15.52:0/1876175595 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9398021070 con 0x7f939c0fbaf0 2023-12-16T20:47:08.229 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.225+0000 7f93927fc700 1 -- 172.21.15.52:0/1876175595 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f939801c400 con 0x7f939c0fbaf0 2023-12-16T20:47:08.229 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.225+0000 7f93927fc700 1 -- 172.21.15.52:0/1876175595 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f939800e070 con 0x7f939c0fbaf0 2023-12-16T20:47:08.229 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.225+0000 7f93a3b5c700 1 -- 172.21.15.52:0/1876175595 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939c10da30 con 0x7f939c0fbaf0 2023-12-16T20:47:08.229 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.225+0000 7f93a3b5c700 1 -- 172.21.15.52:0/1876175595 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f939c10dec0 con 0x7f939c0fbaf0 2023-12-16T20:47:08.229 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.226+0000 7f93a3b5c700 1 -- 172.21.15.52:0/1876175595 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939c04bb80 con 0x7f939c0fbaf0 2023-12-16T20:47:08.230 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.227+0000 7f93927fc700 1 -- 172.21.15.52:0/1876175595 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f939801c950 con 0x7f939c0fbaf0 2023-12-16T20:47:08.230 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.228+0000 7f93927fc700 1 --2- 172.21.15.52:0/1876175595 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f9388068ed0 0x7f938806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:47:08.230 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.228+0000 7f93927fc700 1 -- 172.21.15.52:0/1876175595 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f9398087000 con 0x7f939c0fbaf0 2023-12-16T20:47:08.230 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.228+0000 7f93a10f7700 1 --2- 172.21.15.52:0/1876175595 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f9388068ed0 0x7f938806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:47:08.230 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.228+0000 7f93a10f7700 1 --2- 172.21.15.52:0/1876175595 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f9388068ed0 0x7f938806b380 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f938c0041d0 tx=0x7f938c00b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:47:08.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.232+0000 7f93927fc700 1 -- 172.21.15.52:0/1876175595 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9398057de0 con 0x7f939c0fbaf0 2023-12-16T20:47:08.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:07 smithi150 ceph-mon[88000]: from='osd.0 [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2023-12-16T20:47:08.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:07 smithi150 ceph-mon[88000]: osdmap e8: 1 total, 0 up, 1 in 2023-12-16T20:47:08.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:07 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T20:47:08.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:07 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T20:47:08.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:07 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:08.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:07 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:08.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:07 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:47:08.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:07 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:08.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:07 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:08.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:07 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:47:08.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:07 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:08.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:07 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:47:08.446 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:08.445+0000 7f93a3b5c700 1 -- 172.21.15.52:0/1876175595 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f939c10e320 con 0x7f9388068ed0 2023-12-16T20:47:09.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:08 smithi052 ceph-mon[80761]: purged_snaps scrub starts 2023-12-16T20:47:09.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:08 smithi052 ceph-mon[80761]: purged_snaps scrub ok 2023-12-16T20:47:09.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:08 smithi052 ceph-mon[80761]: Detected new or changed devices on smithi052 2023-12-16T20:47:09.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:08 smithi052 ceph-mon[80761]: Adjusting osd_memory_target on smithi052 to 12869M 2023-12-16T20:47:09.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:08 smithi052 ceph-mon[80761]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:47:09.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:08 smithi052 ceph-mon[80761]: osd.0 [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775] boot 2023-12-16T20:47:09.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:08 smithi052 ceph-mon[80761]: osdmap e9: 1 total, 1 up, 1 in 2023-12-16T20:47:09.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:08 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T20:47:09.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:08 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T20:47:09.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:08 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T20:47:09.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:08 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:09.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:08 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:09.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:08 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:09.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:08 smithi150 ceph-mon[88000]: purged_snaps scrub starts 2023-12-16T20:47:09.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:08 smithi150 ceph-mon[88000]: purged_snaps scrub ok 2023-12-16T20:47:09.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:08 smithi150 ceph-mon[88000]: Detected new or changed devices on smithi052 2023-12-16T20:47:09.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:08 smithi150 ceph-mon[88000]: Adjusting osd_memory_target on smithi052 to 12869M 2023-12-16T20:47:09.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:08 smithi150 ceph-mon[88000]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T20:47:09.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:08 smithi150 ceph-mon[88000]: osd.0 [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775] boot 2023-12-16T20:47:09.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:08 smithi150 ceph-mon[88000]: osdmap e9: 1 total, 1 up, 1 in 2023-12-16T20:47:09.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:08 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T20:47:09.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:08 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T20:47:09.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:08 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T20:47:09.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:08 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:09.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:08 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:09.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:08 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:10.196 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:09 smithi052 ceph-mon[80761]: from='client.24111 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:47:10.196 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:09 smithi052 ceph-mon[80761]: osdmap e10: 1 total, 1 up, 1 in 2023-12-16T20:47:10.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:09 smithi150 ceph-mon[88000]: from='client.24111 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:47:10.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:09 smithi150 ceph-mon[88000]: osdmap e10: 1 total, 1 up, 1 in 2023-12-16T20:47:11.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:10 smithi052 ceph-mon[80761]: pgmap v24: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:11.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:10 smithi150 ceph-mon[88000]: pgmap v24: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:12.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:11 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/886569033' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cf65e84-7be8-4d6c-872d-d908e7a55c46"}]: dispatch 2023-12-16T20:47:12.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:11 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/886569033' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8cf65e84-7be8-4d6c-872d-d908e7a55c46"}]': finished 2023-12-16T20:47:12.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:11 smithi052 ceph-mon[80761]: osdmap e11: 2 total, 1 up, 2 in 2023-12-16T20:47:12.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:11 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T20:47:12.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:11 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:12.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:11 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:12.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:11 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:12.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:11 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:47:12.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:11 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:12.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:11 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/1053369024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T20:47:12.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:11 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/886569033' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cf65e84-7be8-4d6c-872d-d908e7a55c46"}]: dispatch 2023-12-16T20:47:12.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:11 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/886569033' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8cf65e84-7be8-4d6c-872d-d908e7a55c46"}]': finished 2023-12-16T20:47:12.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:11 smithi150 ceph-mon[88000]: osdmap e11: 2 total, 1 up, 2 in 2023-12-16T20:47:12.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:11 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T20:47:12.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:11 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:12.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:11 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:12.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:11 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:12.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:11 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:47:12.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:11 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:12.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:11 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/1053369024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T20:47:13.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:12 smithi052 ceph-mon[80761]: pgmap v26: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:13.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:12 smithi150 ceph-mon[88000]: pgmap v26: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:15.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:14 smithi150 ceph-mon[88000]: pgmap v27: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:15.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:14 smithi052 ceph-mon[80761]: pgmap v27: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:17.245 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:16 smithi052 ceph-mon[80761]: pgmap v28: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:17.245 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:16 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-16T20:47:17.245 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:16 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:17.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:16 smithi150 ceph-mon[88000]: pgmap v28: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:17.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:16 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-16T20:47:17.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:16 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:18.108 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:17 smithi052 ceph-mon[80761]: Deploying daemon osd.1 on smithi052 2023-12-16T20:47:18.108 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:17 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:47:18.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:17 smithi150 ceph-mon[88000]: Deploying daemon osd.1 on smithi052 2023-12-16T20:47:18.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:17 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:47:19.244 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:18 smithi052 ceph-mon[80761]: pgmap v29: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:19.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:18 smithi150 ceph-mon[88000]: pgmap v29: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:21.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:20 smithi052 ceph-mon[80761]: pgmap v30: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:21.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:20 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:21.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:20 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:21.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:20 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:47:21.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:20 smithi150 ceph-mon[88000]: pgmap v30: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:21.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:20 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:21.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:20 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:21.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:20 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:47:22.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:22.489+0000 7f93927fc700 1 -- 172.21.15.52:0/1876175595 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f939c10e320 con 0x7f9388068ed0 2023-12-16T20:47:22.490 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 1 on host 'smithi052' 2023-12-16T20:47:22.492 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:22.491+0000 7f93a3b5c700 1 -- 172.21.15.52:0/1876175595 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f9388068ed0 msgr2=0x7f938806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:47:22.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:22.491+0000 7f93a3b5c700 1 --2- 172.21.15.52:0/1876175595 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f9388068ed0 0x7f938806b380 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f938c0041d0 tx=0x7f938c00b040 comp rx=0 tx=0).stop 2023-12-16T20:47:22.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:22.491+0000 7f93a3b5c700 1 -- 172.21.15.52:0/1876175595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f939c0fbaf0 msgr2=0x7f939c10c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:47:22.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:22.491+0000 7f93a3b5c700 1 --2- 172.21.15.52:0/1876175595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f939c0fbaf0 0x7f939c10c3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9398000c00 tx=0x7f9398009e30 comp rx=0 tx=0).stop 2023-12-16T20:47:22.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:22.491+0000 7f93a3b5c700 1 -- 172.21.15.52:0/1876175595 shutdown_connections 2023-12-16T20:47:22.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:22.491+0000 7f93a3b5c700 1 --2- 172.21.15.52:0/1876175595 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f9388068ed0 0x7f938806b380 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:22.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:22.491+0000 7f93a3b5c700 1 --2- 172.21.15.52:0/1876175595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f939c0fbaf0 0x7f939c10c3d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:22.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:22.491+0000 7f93a3b5c700 1 --2- 172.21.15.52:0/1876175595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f939c0fcd00 0x7f939c10c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:22.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:22.492+0000 7f93a3b5c700 1 -- 172.21.15.52:0/1876175595 >> 172.21.15.52:0/1876175595 conn(0x7f939c0f71d0 msgr2=0x7f939c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:47:22.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:22.492+0000 7f93a3b5c700 1 -- 172.21.15.52:0/1876175595 shutdown_connections 2023-12-16T20:47:22.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:22.492+0000 7f93a3b5c700 1 -- 172.21.15.52:0/1876175595 wait complete. 2023-12-16T20:47:23.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:22 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:23.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:22 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:23.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:22 smithi052 ceph-mon[80761]: pgmap v31: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:22 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:22 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:23.102 DEBUG:teuthology.orchestra.run.smithi052:osd.1> sudo journalctl -f -n 0 -u ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@osd.1.service 2023-12-16T20:47:23.104 INFO:tasks.cephadm:Deploying osd.2 on smithi052 with /dev/vg_nvme/lv_2... 2023-12-16T20:47:23.104 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-12-16T20:47:23.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:22 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:23.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:22 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:23.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:22 smithi150 ceph-mon[88000]: pgmap v31: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:23.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:22 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:23.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:22 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:23.691 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:47:24.702 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:47:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:47:24.461+0000 7fe370156540 -1 osd.1 0 log_to_monitors true 2023-12-16T20:47:25.098 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-12-16T20:47:25.098 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-12-16T20:47:25.098 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2023-12-16T20:47:25.098 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2023-12-16T20:47:25.099 INFO:teuthology.orchestra.run.smithi052.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00978151 s, 1.1 GB/s 2023-12-16T20:47:25.099 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2023-12-16T20:47:25.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:24 smithi052 ceph-mon[80761]: pgmap v32: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:24 smithi052 ceph-mon[80761]: from='osd.1 [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-16T20:47:25.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:24 smithi150 ceph-mon[88000]: pgmap v32: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:25.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:24 smithi150 ceph-mon[88000]: from='osd.1 [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-16T20:47:25.477 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi052:vg_nvme/lv_2 2023-12-16T20:47:25.757 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:47:26.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:47:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:47:25.956+0000 7fe36693b700 -1 osd.1 0 waiting for initial osdmap 2023-12-16T20:47:26.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:47:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:47:25.961+0000 7fe361736700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T20:47:26.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:25 smithi052 ceph-mon[80761]: from='osd.1 [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-16T20:47:26.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:25 smithi052 ceph-mon[80761]: osdmap e12: 2 total, 1 up, 2 in 2023-12-16T20:47:26.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:25 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T20:47:26.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:25 smithi052 ceph-mon[80761]: from='osd.1 [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2023-12-16T20:47:26.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:25 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:26.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:25 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:26.253 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:25 smithi150 ceph-mon[88000]: from='osd.1 [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-16T20:47:26.254 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:25 smithi150 ceph-mon[88000]: osdmap e12: 2 total, 1 up, 2 in 2023-12-16T20:47:26.254 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:25 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T20:47:26.254 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:25 smithi150 ceph-mon[88000]: from='osd.1 [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2023-12-16T20:47:26.254 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:25 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:26.254 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:25 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:26.644 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.642+0000 7fab53916700 1 -- 172.21.15.52:0/717613311 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fab4c0fcd00 msgr2=0x7fab4c0fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:47:26.644 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.642+0000 7fab53916700 1 --2- 172.21.15.52:0/717613311 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fab4c0fcd00 0x7fab4c0fd150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fab40004940 tx=0x7fab40009e30 comp rx=0 tx=0).stop 2023-12-16T20:47:26.644 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.642+0000 7fab53916700 1 -- 172.21.15.52:0/717613311 shutdown_connections 2023-12-16T20:47:26.644 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.642+0000 7fab53916700 1 --2- 172.21.15.52:0/717613311 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fab4c0fcd00 0x7fab4c0fd150 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:26.644 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.642+0000 7fab53916700 1 --2- 172.21.15.52:0/717613311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab4c0fbaf0 0x7fab4c0fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:26.644 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.642+0000 7fab53916700 1 -- 172.21.15.52:0/717613311 >> 172.21.15.52:0/717613311 conn(0x7fab4c0f7190 msgr2=0x7fab4c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:47:26.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.642+0000 7fab53916700 1 -- 172.21.15.52:0/717613311 shutdown_connections 2023-12-16T20:47:26.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.642+0000 7fab53916700 1 -- 172.21.15.52:0/717613311 wait complete. 2023-12-16T20:47:26.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.643+0000 7fab53916700 1 Processor -- start 2023-12-16T20:47:26.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.643+0000 7fab53916700 1 -- start start 2023-12-16T20:47:26.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.643+0000 7fab53916700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab4c0fbaf0 0x7fab4c0ffbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:47:26.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.643+0000 7fab53916700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fab4c0fcd00 0x7fab4c0fe040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:47:26.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.643+0000 7fab53916700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c0fe640 con 0x7fab4c0fcd00 2023-12-16T20:47:26.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.643+0000 7fab53916700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c0fe7b0 con 0x7fab4c0fbaf0 2023-12-16T20:47:26.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.643+0000 7fab50eb1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fab4c0fcd00 0x7fab4c0fe040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:47:26.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.643+0000 7fab50eb1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fab4c0fcd00 0x7fab4c0fe040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41932/0 (socket says 172.21.15.52:41932) 2023-12-16T20:47:26.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.643+0000 7fab50eb1700 1 -- 172.21.15.52:0/240427361 learned_addr learned my addr 172.21.15.52:0/240427361 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:47:26.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.643+0000 7fab516b2700 1 --2- 172.21.15.52:0/240427361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab4c0fbaf0 0x7fab4c0ffbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:47:26.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.643+0000 7fab50eb1700 1 -- 172.21.15.52:0/240427361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab4c0fbaf0 msgr2=0x7fab4c0ffbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:47:26.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.643+0000 7fab50eb1700 1 --2- 172.21.15.52:0/240427361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab4c0fbaf0 0x7fab4c0ffbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:26.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.643+0000 7fab50eb1700 1 -- 172.21.15.52:0/240427361 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab40005040 con 0x7fab4c0fcd00 2023-12-16T20:47:26.646 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.644+0000 7fab50eb1700 1 --2- 172.21.15.52:0/240427361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fab4c0fcd00 0x7fab4c0fe040 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fab4000b3b0 tx=0x7fab40009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:47:26.647 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.644+0000 7fab3dffb700 1 -- 172.21.15.52:0/240427361 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab4000e040 con 0x7fab4c0fcd00 2023-12-16T20:47:26.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.644+0000 7fab53916700 1 -- 172.21.15.52:0/240427361 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4c0fea30 con 0x7fab4c0fcd00 2023-12-16T20:47:26.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.644+0000 7fab3dffb700 1 -- 172.21.15.52:0/240427361 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab4000bcd0 con 0x7fab4c0fcd00 2023-12-16T20:47:26.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.644+0000 7fab3dffb700 1 -- 172.21.15.52:0/240427361 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab4001c630 con 0x7fab4c0fcd00 2023-12-16T20:47:26.649 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.644+0000 7fab53916700 1 -- 172.21.15.52:0/240427361 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4c1155c0 con 0x7fab4c0fcd00 2023-12-16T20:47:26.649 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.645+0000 7fab53916700 1 -- 172.21.15.52:0/240427361 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4c04bb80 con 0x7fab4c0fcd00 2023-12-16T20:47:26.649 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.645+0000 7fab3dffb700 1 -- 172.21.15.52:0/240427361 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7fab40013420 con 0x7fab4c0fcd00 2023-12-16T20:47:26.649 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.646+0000 7fab3dffb700 1 --2- 172.21.15.52:0/240427361 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fab38068ed0 0x7fab3806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:47:26.649 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.646+0000 7fab3dffb700 1 -- 172.21.15.52:0/240427361 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2130+0+0 (secure 0 0 0) 0x7fab400873a0 con 0x7fab4c0fcd00 2023-12-16T20:47:26.649 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.646+0000 7fab516b2700 1 --2- 172.21.15.52:0/240427361 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fab38068ed0 0x7fab3806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:47:26.650 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.646+0000 7fab516b2700 1 --2- 172.21.15.52:0/240427361 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fab38068ed0 0x7fab3806b380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fab480041d0 tx=0x7fab48009040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:47:26.651 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.651+0000 7fab3dffb700 1 -- 172.21.15.52:0/240427361 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fab40058190 con 0x7fab4c0fcd00 2023-12-16T20:47:26.882 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:26.881+0000 7fab53916700 1 -- 172.21.15.52:0/240427361 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fab4c065a80 con 0x7fab38068ed0 2023-12-16T20:47:27.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:26 smithi052 ceph-mon[80761]: pgmap v34: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:27.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:26 smithi052 ceph-mon[80761]: from='osd.1 [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2023-12-16T20:47:27.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:26 smithi052 ceph-mon[80761]: osdmap e13: 2 total, 1 up, 2 in 2023-12-16T20:47:27.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T20:47:27.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T20:47:27.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T20:47:27.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T20:47:27.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:27.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:26 smithi150 ceph-mon[88000]: pgmap v34: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:27.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:26 smithi150 ceph-mon[88000]: from='osd.1 [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2023-12-16T20:47:27.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:26 smithi150 ceph-mon[88000]: osdmap e13: 2 total, 1 up, 2 in 2023-12-16T20:47:27.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:26 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T20:47:27.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:26 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T20:47:27.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:26 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T20:47:27.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:26 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T20:47:27.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:26 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:28.127 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:27 smithi052 ceph-mon[80761]: purged_snaps scrub starts 2023-12-16T20:47:28.127 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:27 smithi052 ceph-mon[80761]: purged_snaps scrub ok 2023-12-16T20:47:28.127 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:27 smithi052 ceph-mon[80761]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:47:28.127 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:27 smithi052 ceph-mon[80761]: osd.1 [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721] boot 2023-12-16T20:47:28.127 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:27 smithi052 ceph-mon[80761]: osdmap e14: 2 total, 2 up, 2 in 2023-12-16T20:47:28.127 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:27 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T20:47:28.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:27 smithi150 ceph-mon[88000]: purged_snaps scrub starts 2023-12-16T20:47:28.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:27 smithi150 ceph-mon[88000]: purged_snaps scrub ok 2023-12-16T20:47:28.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:27 smithi150 ceph-mon[88000]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:47:28.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:27 smithi150 ceph-mon[88000]: osd.1 [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721] boot 2023-12-16T20:47:28.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:27 smithi150 ceph-mon[88000]: osdmap e14: 2 total, 2 up, 2 in 2023-12-16T20:47:28.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:27 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T20:47:29.118 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:28 smithi052 ceph-mon[80761]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:29.118 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:28 smithi052 ceph-mon[80761]: osdmap e15: 2 total, 2 up, 2 in 2023-12-16T20:47:29.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:28 smithi150 ceph-mon[88000]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T20:47:29.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:28 smithi150 ceph-mon[88000]: osdmap e15: 2 total, 2 up, 2 in 2023-12-16T20:47:30.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: Detected new or changed devices on smithi052 2023-12-16T20:47:30.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:30.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:30.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:47:30.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:47:30.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: Adjusting osd_memory_target on smithi052 to 6434M 2023-12-16T20:47:30.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:30.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:30.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:47:30.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:30.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:47:30.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: pgmap v39: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:30.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/3452041867' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd375c62-57d0-4899-aa93-b582a664abbd"}]: dispatch 2023-12-16T20:47:30.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/3452041867' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd375c62-57d0-4899-aa93-b582a664abbd"}]': finished 2023-12-16T20:47:30.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: osdmap e16: 3 total, 2 up, 3 in 2023-12-16T20:47:30.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:30 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T20:47:30.470 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: Detected new or changed devices on smithi052 2023-12-16T20:47:30.470 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:30.470 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:30.470 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:47:30.471 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:47:30.471 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: Adjusting osd_memory_target on smithi052 to 6434M 2023-12-16T20:47:30.471 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:30.471 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:30.471 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:47:30.471 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:30.471 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:47:30.471 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: pgmap v39: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:30.471 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3452041867' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd375c62-57d0-4899-aa93-b582a664abbd"}]: dispatch 2023-12-16T20:47:30.471 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3452041867' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd375c62-57d0-4899-aa93-b582a664abbd"}]': finished 2023-12-16T20:47:30.471 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: osdmap e16: 3 total, 2 up, 3 in 2023-12-16T20:47:30.471 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:30 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T20:47:31.468 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:31 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/4246727883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T20:47:31.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:31 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/4246727883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T20:47:32.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:32 smithi052 ceph-mon[80761]: pgmap v41: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:32.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:32 smithi150 ceph-mon[88000]: pgmap v41: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:33.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:33 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:33.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:33 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:33.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:33 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:33.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:33 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:47:33.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:33 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:33.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:33 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:47:33.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:33 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:33.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:33 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:33.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:33 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:33.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:33 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:47:33.689 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:33 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:33.689 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:33 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:47:34.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:34 smithi052 ceph-mon[80761]: pgmap v42: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:34.589 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:34 smithi150 ceph-mon[88000]: pgmap v42: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:36.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:35 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-16T20:47:36.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:35 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:36.253 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:35 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-16T20:47:36.253 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:35 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:37.151 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:36 smithi052 ceph-mon[80761]: pgmap v43: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:37.151 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:36 smithi052 ceph-mon[80761]: Deploying daemon osd.2 on smithi052 2023-12-16T20:47:37.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:36 smithi150 ceph-mon[88000]: pgmap v43: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:37.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:36 smithi150 ceph-mon[88000]: Deploying daemon osd.2 on smithi052 2023-12-16T20:47:38.518 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:38 smithi052 ceph-mon[80761]: pgmap v44: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:38.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:38 smithi150 ceph-mon[88000]: pgmap v44: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:40.905 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:40 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:40.906 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:40 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:40.906 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:40 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:47:40.906 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:40 smithi052 ceph-mon[80761]: pgmap v45: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:40.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:40 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:40.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:40 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:40.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:40 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:47:40.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:40 smithi150 ceph-mon[88000]: pgmap v45: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:41.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:41.525+0000 7fab3dffb700 1 -- 172.21.15.52:0/240427361 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fab4c065a80 con 0x7fab38068ed0 2023-12-16T20:47:41.526 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 2 on host 'smithi052' 2023-12-16T20:47:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:41.527+0000 7fab53916700 1 -- 172.21.15.52:0/240427361 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fab38068ed0 msgr2=0x7fab3806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:47:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:41.527+0000 7fab53916700 1 --2- 172.21.15.52:0/240427361 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fab38068ed0 0x7fab3806b380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fab480041d0 tx=0x7fab48009040 comp rx=0 tx=0).stop 2023-12-16T20:47:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:41.527+0000 7fab53916700 1 -- 172.21.15.52:0/240427361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fab4c0fcd00 msgr2=0x7fab4c0fe040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:47:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:41.527+0000 7fab53916700 1 --2- 172.21.15.52:0/240427361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fab4c0fcd00 0x7fab4c0fe040 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fab4000b3b0 tx=0x7fab40009e30 comp rx=0 tx=0).stop 2023-12-16T20:47:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:41.528+0000 7fab53916700 1 -- 172.21.15.52:0/240427361 shutdown_connections 2023-12-16T20:47:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:41.528+0000 7fab53916700 1 --2- 172.21.15.52:0/240427361 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fab38068ed0 0x7fab3806b380 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:41.528+0000 7fab53916700 1 --2- 172.21.15.52:0/240427361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab4c0fbaf0 0x7fab4c0ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:41.528+0000 7fab53916700 1 --2- 172.21.15.52:0/240427361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fab4c0fcd00 0x7fab4c0fe040 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:41.528+0000 7fab53916700 1 -- 172.21.15.52:0/240427361 >> 172.21.15.52:0/240427361 conn(0x7fab4c0f7190 msgr2=0x7fab4c065370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:47:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:41.528+0000 7fab53916700 1 -- 172.21.15.52:0/240427361 shutdown_connections 2023-12-16T20:47:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:47:41.528+0000 7fab53916700 1 -- 172.21.15.52:0/240427361 wait complete. 2023-12-16T20:47:41.975 DEBUG:teuthology.orchestra.run.smithi052:osd.2> sudo journalctl -f -n 0 -u ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@osd.2.service 2023-12-16T20:47:41.978 INFO:tasks.cephadm:Deploying osd.3 on smithi150 with /dev/vg_nvme/lv_4... 2023-12-16T20:47:41.978 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-12-16T20:47:42.236 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi150/config 2023-12-16T20:47:42.654 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:42 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:42.654 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:42 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:42.655 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:42 smithi150 ceph-mon[88000]: pgmap v46: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:42.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:42 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:42.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:42 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:42.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:42 smithi052 ceph-mon[80761]: pgmap v46: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:43.285 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-12-16T20:47:43.285 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-12-16T20:47:43.285 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2023-12-16T20:47:43.285 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2023-12-16T20:47:43.285 INFO:teuthology.orchestra.run.smithi150.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113821 s, 921 MB/s 2023-12-16T20:47:43.286 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2023-12-16T20:47:43.587 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_4 2023-12-16T20:47:43.838 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi150/config 2023-12-16T20:47:43.904 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:43 smithi150 ceph-mon[88000]: from='osd.2 [v2:172.21.15.52:6818/4281298930,v1:172.21.15.52:6819/4281298930]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-16T20:47:43.904 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:43 smithi150 ceph-mon[88000]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-16T20:47:44.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:43 smithi052 ceph-mon[80761]: from='osd.2 [v2:172.21.15.52:6818/4281298930,v1:172.21.15.52:6819/4281298930]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-16T20:47:44.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:43 smithi052 ceph-mon[80761]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-16T20:47:44.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:47:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:47:43.621+0000 7f0978da4540 -1 osd.2 0 log_to_monitors true 2023-12-16T20:47:44.457 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.455+0000 7f8d25b62700 1 -- 172.21.15.150:0/1079030489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d200fe270 msgr2=0x7f8d200fe680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:47:44.457 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.455+0000 7f8d25b62700 1 --2- 172.21.15.150:0/1079030489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d200fe270 0x7f8d200fe680 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8d14004940 tx=0x7f8d14009e30 comp rx=0 tx=0).stop 2023-12-16T20:47:44.457 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.455+0000 7f8d25b62700 1 -- 172.21.15.150:0/1079030489 shutdown_connections 2023-12-16T20:47:44.457 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.455+0000 7f8d25b62700 1 --2- 172.21.15.150:0/1079030489 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d200fec50 0x7f8d200fad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:44.457 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.455+0000 7f8d25b62700 1 --2- 172.21.15.150:0/1079030489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d200fe270 0x7f8d200fe680 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:44.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.455+0000 7f8d25b62700 1 -- 172.21.15.150:0/1079030489 >> 172.21.15.150:0/1079030489 conn(0x7f8d200f6950 msgr2=0x7f8d200f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:47:44.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.455+0000 7f8d25b62700 1 -- 172.21.15.150:0/1079030489 shutdown_connections 2023-12-16T20:47:44.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.455+0000 7f8d25b62700 1 -- 172.21.15.150:0/1079030489 wait complete. 2023-12-16T20:47:44.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.455+0000 7f8d25b62700 1 Processor -- start 2023-12-16T20:47:44.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.455+0000 7f8d25b62700 1 -- start start 2023-12-16T20:47:44.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d25b62700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d200fe270 0x7f8d200fdab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:47:44.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d25b62700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d200fbf30 0x7f8d200fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:47:44.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d25b62700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d200fc930 con 0x7f8d200fbf30 2023-12-16T20:47:44.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d25b62700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d200fca70 con 0x7f8d200fe270 2023-12-16T20:47:44.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d1f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d200fe270 0x7f8d200fdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:47:44.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d1f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d200fe270 0x7f8d200fdab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37126/0 (socket says 172.21.15.150:37126) 2023-12-16T20:47:44.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d1f7fe700 1 -- 172.21.15.150:0/3266500117 learned_addr learned my addr 172.21.15.150:0/3266500117 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:47:44.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d1effd700 1 --2- 172.21.15.150:0/3266500117 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d200fbf30 0x7f8d200fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:47:44.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d1f7fe700 1 -- 172.21.15.150:0/3266500117 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d200fbf30 msgr2=0x7f8d200fc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:47:44.459 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d1f7fe700 1 --2- 172.21.15.150:0/3266500117 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d200fbf30 0x7f8d200fc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:44.459 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d1f7fe700 1 -- 172.21.15.150:0/3266500117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d14005040 con 0x7f8d200fe270 2023-12-16T20:47:44.459 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d1f7fe700 1 --2- 172.21.15.150:0/3266500117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d200fe270 0x7f8d200fdab0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8d14009fd0 tx=0x7f8d14009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:47:44.459 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d07fff700 1 -- 172.21.15.150:0/3266500117 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d1400e040 con 0x7f8d200fe270 2023-12-16T20:47:44.459 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d25b62700 1 -- 172.21.15.150:0/3266500117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d200fccf0 con 0x7f8d200fe270 2023-12-16T20:47:44.459 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d07fff700 1 -- 172.21.15.150:0/3266500117 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d14013420 con 0x7f8d200fe270 2023-12-16T20:47:44.459 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d07fff700 1 -- 172.21.15.150:0/3266500117 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d1401c630 con 0x7f8d200fe270 2023-12-16T20:47:44.459 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.456+0000 7f8d25b62700 1 -- 172.21.15.150:0/3266500117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d201158e0 con 0x7f8d200fe270 2023-12-16T20:47:44.460 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.458+0000 7f8d25b62700 1 -- 172.21.15.150:0/3266500117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2004bb80 con 0x7f8d200fe270 2023-12-16T20:47:44.460 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.458+0000 7f8d07fff700 1 -- 172.21.15.150:0/3266500117 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f8d14013a00 con 0x7f8d200fe270 2023-12-16T20:47:44.460 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.458+0000 7f8d07fff700 1 --2- 172.21.15.150:0/3266500117 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8d08068ed0 0x7f8d0806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:47:44.460 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.458+0000 7f8d07fff700 1 -- 172.21.15.150:0/3266500117 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 2546+0+0 (secure 0 0 0) 0x7f8d140873c0 con 0x7f8d200fe270 2023-12-16T20:47:44.460 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.459+0000 7f8d1effd700 1 --2- 172.21.15.150:0/3266500117 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8d08068ed0 0x7f8d0806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:47:44.460 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.459+0000 7f8d1effd700 1 --2- 172.21.15.150:0/3266500117 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8d08068ed0 0x7f8d0806b380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009fd0 tx=0x7f8d1000b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:47:44.463 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.463+0000 7f8d07fff700 1 -- 172.21.15.150:0/3266500117 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8d14057f60 con 0x7f8d200fe270 2023-12-16T20:47:44.661 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:44.660+0000 7f8d25b62700 1 -- 172.21.15.150:0/3266500117 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d20102970 con 0x7f8d08068ed0 2023-12-16T20:47:44.727 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:44 smithi150 ceph-mon[88000]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-16T20:47:44.727 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:44 smithi150 ceph-mon[88000]: osdmap e17: 3 total, 2 up, 3 in 2023-12-16T20:47:44.727 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:44 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T20:47:44.727 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:44 smithi150 ceph-mon[88000]: from='osd.2 [v2:172.21.15.52:6818/4281298930,v1:172.21.15.52:6819/4281298930]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2023-12-16T20:47:44.727 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:44 smithi150 ceph-mon[88000]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2023-12-16T20:47:44.727 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:44 smithi150 ceph-mon[88000]: pgmap v48: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:44.727 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:44 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:44.727 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:44 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:44.727 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:44 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T20:47:44.727 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:44 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T20:47:44.727 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:44 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:44.966 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:47:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:47:44.687+0000 7f096f589700 -1 osd.2 0 waiting for initial osdmap 2023-12-16T20:47:44.967 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:47:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:47:44.692+0000 7f096ab85700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T20:47:44.967 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:44 smithi052 ceph-mon[80761]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-16T20:47:44.967 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:44 smithi052 ceph-mon[80761]: osdmap e17: 3 total, 2 up, 3 in 2023-12-16T20:47:44.967 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:44 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T20:47:44.967 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:44 smithi052 ceph-mon[80761]: from='osd.2 [v2:172.21.15.52:6818/4281298930,v1:172.21.15.52:6819/4281298930]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2023-12-16T20:47:44.967 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:44 smithi052 ceph-mon[80761]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2023-12-16T20:47:44.967 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:44 smithi052 ceph-mon[80761]: pgmap v48: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:44.967 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:44 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:44.967 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:44 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:44.967 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:44 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T20:47:44.967 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:44 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T20:47:44.968 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:44 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:45.969 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:45 smithi052 ceph-mon[80761]: from='client.24123 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:47:45.969 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:45 smithi052 ceph-mon[80761]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2023-12-16T20:47:45.969 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:45 smithi052 ceph-mon[80761]: osdmap e18: 3 total, 2 up, 3 in 2023-12-16T20:47:45.969 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:45 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T20:47:45.969 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:45 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T20:47:46.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:45 smithi150 ceph-mon[88000]: from='client.24123 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:47:46.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:45 smithi150 ceph-mon[88000]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2023-12-16T20:47:46.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:45 smithi150 ceph-mon[88000]: osdmap e18: 3 total, 2 up, 3 in 2023-12-16T20:47:46.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:45 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T20:47:46.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:45 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T20:47:47.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: purged_snaps scrub starts 2023-12-16T20:47:47.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: purged_snaps scrub ok 2023-12-16T20:47:47.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: osd.2 [v2:172.21.15.52:6818/4281298930,v1:172.21.15.52:6819/4281298930] boot 2023-12-16T20:47:47.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: osdmap e19: 3 total, 3 up, 3 in 2023-12-16T20:47:47.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T20:47:47.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: pgmap v51: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:47.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-12-16T20:47:47.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/3771096041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31298367-7548-4ef5-bd2b-a6f586ec14a7"}]: dispatch 2023-12-16T20:47:47.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31298367-7548-4ef5-bd2b-a6f586ec14a7"}]: dispatch 2023-12-16T20:47:47.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-12-16T20:47:47.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31298367-7548-4ef5-bd2b-a6f586ec14a7"}]': finished 2023-12-16T20:47:47.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: osdmap e20: 4 total, 3 up, 4 in 2023-12-16T20:47:47.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T20:47:47.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-12-16T20:47:47.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:47.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:47.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:47:47.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:47:47.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:47:47.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:47.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:47.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:47:47.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:47.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:46 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:47:47.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: purged_snaps scrub starts 2023-12-16T20:47:47.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: purged_snaps scrub ok 2023-12-16T20:47:47.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: osd.2 [v2:172.21.15.52:6818/4281298930,v1:172.21.15.52:6819/4281298930] boot 2023-12-16T20:47:47.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: osdmap e19: 3 total, 3 up, 3 in 2023-12-16T20:47:47.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T20:47:47.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: pgmap v51: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T20:47:47.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-12-16T20:47:47.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/3771096041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31298367-7548-4ef5-bd2b-a6f586ec14a7"}]: dispatch 2023-12-16T20:47:47.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31298367-7548-4ef5-bd2b-a6f586ec14a7"}]: dispatch 2023-12-16T20:47:47.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-12-16T20:47:47.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31298367-7548-4ef5-bd2b-a6f586ec14a7"}]': finished 2023-12-16T20:47:47.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: osdmap e20: 4 total, 3 up, 4 in 2023-12-16T20:47:47.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T20:47:47.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-12-16T20:47:47.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:47.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:47.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:47:47.190 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:47:47.190 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:47:47.190 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:47.190 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:47.190 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:47:47.190 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:47.190 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:46 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:47:47.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:47 smithi052 ceph-mon[80761]: Detected new or changed devices on smithi052 2023-12-16T20:47:47.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:47 smithi052 ceph-mon[80761]: Adjusting osd_memory_target on smithi052 to 4289M 2023-12-16T20:47:47.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:47 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/850063543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T20:47:47.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:47 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-12-16T20:47:47.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:47 smithi052 ceph-mon[80761]: osdmap e21: 4 total, 3 up, 4 in 2023-12-16T20:47:47.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:47 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T20:47:48.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:47 smithi150 ceph-mon[88000]: Detected new or changed devices on smithi052 2023-12-16T20:47:48.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:47 smithi150 ceph-mon[88000]: Adjusting osd_memory_target on smithi052 to 4289M 2023-12-16T20:47:48.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:47 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/850063543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T20:47:48.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:47 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-12-16T20:47:48.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:47 smithi150 ceph-mon[88000]: osdmap e21: 4 total, 3 up, 4 in 2023-12-16T20:47:48.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:47 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T20:47:48.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:47:47 smithi052 sudo[119514]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-16T20:47:48.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:47:47 smithi052 sudo[119514]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T20:47:48.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:47:47 smithi052 sudo[119514]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T20:47:48.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:47:47 smithi052 sudo[119514]: pam_unix(sudo:session): session closed for user root 2023-12-16T20:47:48.703 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:47:48 smithi052 sudo[119597]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-16T20:47:48.704 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:47:48 smithi052 sudo[119597]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T20:47:48.704 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:47:48 smithi052 sudo[119597]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T20:47:48.704 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:47:48 smithi052 sudo[119597]: pam_unix(sudo:session): session closed for user root 2023-12-16T20:47:48.706 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:48.706+0000 7f8d07fff700 1 -- 172.21.15.150:0/3266500117 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f8d140548a0 con 0x7f8d200fe270 2023-12-16T20:47:48.989 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:47:48 smithi052 sudo[119676]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-16T20:47:48.989 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:47:48 smithi052 sudo[119676]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T20:47:48.990 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:47:48 smithi052 sudo[119676]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T20:47:48.990 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:47:48 smithi052 sudo[119676]: pam_unix(sudo:session): session closed for user root 2023-12-16T20:47:48.990 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:48 smithi052 ceph-mon[80761]: pgmap v54: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-16T20:47:48.990 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:48 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:47:48.990 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:48 smithi052 ceph-mon[80761]: osdmap e22: 4 total, 3 up, 4 in 2023-12-16T20:47:48.990 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:48 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T20:47:48.990 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:48 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:49.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:48 smithi150 ceph-mon[88000]: pgmap v54: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-16T20:47:49.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:48 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:47:49.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:48 smithi150 ceph-mon[88000]: osdmap e22: 4 total, 3 up, 4 in 2023-12-16T20:47:49.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:48 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T20:47:49.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:48 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:49.705 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:47:49 smithi052 sudo[119792]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-16T20:47:49.705 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:47:49 smithi052 sudo[119792]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T20:47:49.705 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:47:49 smithi052 sudo[119792]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T20:47:49.705 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:47:49 smithi052 sudo[119792]: pam_unix(sudo:session): session closed for user root 2023-12-16T20:47:50.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:47:49 smithi052 sudo[119795]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-16T20:47:50.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:47:49 smithi052 sudo[119795]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T20:47:50.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:47:49 smithi052 sudo[119795]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T20:47:50.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:47:49 smithi052 sudo[119795]: pam_unix(sudo:session): session closed for user root 2023-12-16T20:47:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:49 smithi052 ceph-mon[80761]: mgrmap e19: smithi052.nmkcbs(active, since 90s), standbys: smithi150.vjiupe 2023-12-16T20:47:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:49 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:49 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:49 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:49 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:47:50.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:49 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:50.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:49 smithi150 ceph-mon[88000]: mgrmap e19: smithi052.nmkcbs(active, since 90s), standbys: smithi150.vjiupe 2023-12-16T20:47:50.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:49 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:50.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:49 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:50.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:49 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:50.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:49 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:47:50.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:49 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:50.736 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:47:50 smithi052 sudo[119798]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-16T20:47:50.736 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:47:50 smithi052 sudo[119798]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T20:47:50.736 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:47:50 smithi052 sudo[119798]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T20:47:50.736 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:47:50 smithi052 sudo[119798]: pam_unix(sudo:session): session closed for user root 2023-12-16T20:47:51.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:50 smithi052 sudo[119801]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-16T20:47:51.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:50 smithi052 sudo[119801]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T20:47:51.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:50 smithi052 sudo[119801]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T20:47:51.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:50 smithi052 sudo[119801]: pam_unix(sudo:session): session closed for user root 2023-12-16T20:47:51.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:51 smithi052 sudo[119804]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-16T20:47:51.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:51 smithi052 sudo[119804]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T20:47:51.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:51 smithi052 sudo[119804]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T20:47:51.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:51 smithi052 sudo[119804]: pam_unix(sudo:session): session closed for user root 2023-12-16T20:47:51.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:51 smithi052 ceph-mon[80761]: pgmap v56: 1 pgs: 1 active+clean; 449 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2023-12-16T20:47:51.709 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:51 smithi150 sudo[92740]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-16T20:47:51.709 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:51 smithi150 sudo[92740]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T20:47:51.710 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:51 smithi150 sudo[92740]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T20:47:51.710 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:51 smithi150 sudo[92740]: pam_unix(sudo:session): session closed for user root 2023-12-16T20:47:52.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:51 smithi150 sudo[92750]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-16T20:47:52.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:51 smithi150 sudo[92750]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T20:47:52.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:51 smithi150 sudo[92750]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T20:47:52.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:51 smithi150 sudo[92750]: pam_unix(sudo:session): session closed for user root 2023-12-16T20:47:52.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:51 smithi150 ceph-mon[88000]: pgmap v56: 1 pgs: 1 active+clean; 449 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2023-12-16T20:47:52.454 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:52 smithi150 ceph-mon[88000]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T20:47:52.454 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:52 smithi150 ceph-mon[88000]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T20:47:52.455 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:52 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2023-12-16T20:47:52.455 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:52 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:47:52.455 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:52 smithi150 ceph-mon[88000]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T20:47:52.455 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:52 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2023-12-16T20:47:52.455 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:52 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:47:52.455 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:52 smithi150 ceph-mon[88000]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T20:47:52.455 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:52 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:52.455 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:52 smithi150 ceph-mon[88000]: pgmap v57: 1 pgs: 1 active+clean; 449 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2023-12-16T20:47:52.455 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:52 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-16T20:47:52.455 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:52 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:52.455 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:52 smithi150 ceph-mon[88000]: Deploying daemon osd.3 on smithi150 2023-12-16T20:47:52.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:52 smithi052 ceph-mon[80761]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T20:47:52.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:52 smithi052 ceph-mon[80761]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T20:47:52.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:52 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2023-12-16T20:47:52.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:52 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:47:52.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:52 smithi052 ceph-mon[80761]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T20:47:52.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:52 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2023-12-16T20:47:52.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:52 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2023-12-16T20:47:52.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:52 smithi052 ceph-mon[80761]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T20:47:52.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:52 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:52.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:52 smithi052 ceph-mon[80761]: pgmap v57: 1 pgs: 1 active+clean; 449 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2023-12-16T20:47:52.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:52 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-16T20:47:52.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:52 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:47:52.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:52 smithi052 ceph-mon[80761]: Deploying daemon osd.3 on smithi150 2023-12-16T20:47:55.015 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:54 smithi150 ceph-mon[88000]: pgmap v58: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.9 KiB/s rd, 238 KiB/s wr, 16 op/s 2023-12-16T20:47:55.015 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:54 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:55.015 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:54 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:55.015 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:54 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:47:55.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:54 smithi052 ceph-mon[80761]: pgmap v58: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.9 KiB/s rd, 238 KiB/s wr, 16 op/s 2023-12-16T20:47:55.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:54 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:55.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:54 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:55.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:54 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:47:56.855 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 3 on host 'smithi150' 2023-12-16T20:47:56.856 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:56.854+0000 7f8d07fff700 1 -- 172.21.15.150:0/3266500117 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8d20102970 con 0x7f8d08068ed0 2023-12-16T20:47:56.857 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:56.856+0000 7f8d25b62700 1 -- 172.21.15.150:0/3266500117 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8d08068ed0 msgr2=0x7f8d0806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:47:56.857 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:56.856+0000 7f8d25b62700 1 --2- 172.21.15.150:0/3266500117 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8d08068ed0 0x7f8d0806b380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009fd0 tx=0x7f8d1000b040 comp rx=0 tx=0).stop 2023-12-16T20:47:56.857 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:56.856+0000 7f8d25b62700 1 -- 172.21.15.150:0/3266500117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d200fe270 msgr2=0x7f8d200fdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:47:56.857 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:56.856+0000 7f8d25b62700 1 --2- 172.21.15.150:0/3266500117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d200fe270 0x7f8d200fdab0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8d14009fd0 tx=0x7f8d14009e30 comp rx=0 tx=0).stop 2023-12-16T20:47:56.857 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:56.856+0000 7f8d25b62700 1 -- 172.21.15.150:0/3266500117 shutdown_connections 2023-12-16T20:47:56.857 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:56.856+0000 7f8d25b62700 1 --2- 172.21.15.150:0/3266500117 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8d08068ed0 0x7f8d0806b380 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:56.857 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:56.856+0000 7f8d25b62700 1 --2- 172.21.15.150:0/3266500117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d200fe270 0x7f8d200fdab0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:56.858 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:56.856+0000 7f8d25b62700 1 --2- 172.21.15.150:0/3266500117 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d200fbf30 0x7f8d200fc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:47:56.858 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:56.856+0000 7f8d25b62700 1 -- 172.21.15.150:0/3266500117 >> 172.21.15.150:0/3266500117 conn(0x7f8d200f6950 msgr2=0x7f8d20101250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:47:56.858 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:56.857+0000 7f8d25b62700 1 -- 172.21.15.150:0/3266500117 shutdown_connections 2023-12-16T20:47:56.858 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:47:56.857+0000 7f8d25b62700 1 -- 172.21.15.150:0/3266500117 wait complete. 2023-12-16T20:47:57.103 DEBUG:teuthology.orchestra.run.smithi150:osd.3> sudo journalctl -f -n 0 -u ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@osd.3.service 2023-12-16T20:47:57.105 INFO:tasks.cephadm:Deploying osd.4 on smithi150 with /dev/vg_nvme/lv_3... 2023-12-16T20:47:57.105 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-12-16T20:47:57.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:56 smithi150 ceph-mon[88000]: pgmap v59: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.6 KiB/s rd, 174 KiB/s wr, 12 op/s 2023-12-16T20:47:57.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:56 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:57.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:56 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:57.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:56 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:57.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:56 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:57.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:56 smithi052 ceph-mon[80761]: pgmap v59: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.6 KiB/s rd, 174 KiB/s wr, 12 op/s 2023-12-16T20:47:57.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:56 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:57.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:56 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:57.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:56 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:57.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:56 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:47:57.648 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi150/config 2023-12-16T20:47:58.535 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:58 smithi150 ceph-mon[88000]: pgmap v60: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.8 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-12-16T20:47:58.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:58 smithi052 ceph-mon[80761]: pgmap v60: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.8 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-12-16T20:47:58.903 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-12-16T20:47:58.903 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-12-16T20:47:58.903 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2023-12-16T20:47:58.903 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2023-12-16T20:47:58.903 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128453 s, 816 MB/s 2023-12-16T20:47:58.903 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2023-12-16T20:47:59.185 INFO:journalctl@ceph.osd.3.smithi150.stdout:Dec 16 20:47:58 smithi150 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-3[93318]: 2023-12-16T20:47:58.846+0000 7f2cd8eb7540 -1 osd.3 0 log_to_monitors true 2023-12-16T20:47:59.280 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_3 2023-12-16T20:47:59.532 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi150/config 2023-12-16T20:48:00.070 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:59 smithi150 ceph-mon[88000]: Detected new or changed devices on smithi150 2023-12-16T20:48:00.070 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:00.070 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:00.070 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:48:00.070 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:59 smithi150 ceph-mon[88000]: Adjusting osd_memory_target on smithi150 to 15937M 2023-12-16T20:48:00.070 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:00.070 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:00.071 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:48:00.071 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:00.071 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:59 smithi150 ceph-mon[88000]: from='osd.3 [v2:172.21.15.150:6800/2861125970,v1:172.21.15.150:6801/2861125970]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T20:48:00.071 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:59 smithi150 ceph-mon[88000]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T20:48:00.071 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:48:00.071 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:00.071 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:48:00.071 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:47:59 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:00.228 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.226+0000 7f84606fa700 1 -- 172.21.15.150:0/688931833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84580ffd70 msgr2=0x7f84581021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:00.228 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.226+0000 7f84606fa700 1 --2- 172.21.15.150:0/688931833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84580ffd70 0x7f84581021d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8448004940 tx=0x7f8448009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:00.228 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.226+0000 7f84606fa700 1 -- 172.21.15.150:0/688931833 shutdown_connections 2023-12-16T20:48:00.229 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.227+0000 7f84606fa700 1 --2- 172.21.15.150:0/688931833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84580ffd70 0x7f84581021d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:00.229 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.227+0000 7f84606fa700 1 --2- 172.21.15.150:0/688931833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8458066920 0x7f84580ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:00.229 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.227+0000 7f84606fa700 1 -- 172.21.15.150:0/688931833 >> 172.21.15.150:0/688931833 conn(0x7f84580f71d0 msgr2=0x7f84580f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:00.229 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.227+0000 7f84606fa700 1 -- 172.21.15.150:0/688931833 shutdown_connections 2023-12-16T20:48:00.229 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.227+0000 7f84606fa700 1 -- 172.21.15.150:0/688931833 wait complete. 2023-12-16T20:48:00.229 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.227+0000 7f84606fa700 1 Processor -- start 2023-12-16T20:48:00.229 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.227+0000 7f84606fa700 1 -- start start 2023-12-16T20:48:00.229 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.227+0000 7f84606fa700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8458066920 0x7f845810d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:00.229 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.227+0000 7f84606fa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84580ffd70 0x7f845810da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:00.229 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.227+0000 7f84606fa700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845810e120 con 0x7f84580ffd70 2023-12-16T20:48:00.229 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.227+0000 7f84606fa700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845810e960 con 0x7f8458066920 2023-12-16T20:48:00.229 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.227+0000 7f845e496700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8458066920 0x7f845810d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:00.230 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.227+0000 7f845e496700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8458066920 0x7f845810d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49272/0 (socket says 172.21.15.150:49272) 2023-12-16T20:48:00.230 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.227+0000 7f845e496700 1 -- 172.21.15.150:0/4007963010 learned_addr learned my addr 172.21.15.150:0/4007963010 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:48:00.230 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.227+0000 7f845dc95700 1 --2- 172.21.15.150:0/4007963010 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84580ffd70 0x7f845810da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:00.230 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.228+0000 7f845e496700 1 -- 172.21.15.150:0/4007963010 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84580ffd70 msgr2=0x7f845810da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:00.230 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.228+0000 7f845e496700 1 --2- 172.21.15.150:0/4007963010 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84580ffd70 0x7f845810da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:00.230 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.228+0000 7f845e496700 1 -- 172.21.15.150:0/4007963010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8448005040 con 0x7f8458066920 2023-12-16T20:48:00.230 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.228+0000 7f845e496700 1 --2- 172.21.15.150:0/4007963010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8458066920 0x7f845810d550 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f845400b790 tx=0x7f8454009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:00.230 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.228+0000 7f844effd700 1 -- 172.21.15.150:0/4007963010 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f845400f770 con 0x7f8458066920 2023-12-16T20:48:00.230 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.228+0000 7f844effd700 1 -- 172.21.15.150:0/4007963010 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f845400fd80 con 0x7f8458066920 2023-12-16T20:48:00.230 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.228+0000 7f844effd700 1 -- 172.21.15.150:0/4007963010 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f845400d310 con 0x7f8458066920 2023-12-16T20:48:00.230 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.228+0000 7f84606fa700 1 -- 172.21.15.150:0/4007963010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845810ec40 con 0x7f8458066920 2023-12-16T20:48:00.230 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.228+0000 7f84606fa700 1 -- 172.21.15.150:0/4007963010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845810f180 con 0x7f8458066920 2023-12-16T20:48:00.231 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.229+0000 7f84606fa700 1 -- 172.21.15.150:0/4007963010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8458107640 con 0x7f8458066920 2023-12-16T20:48:00.231 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.229+0000 7f844effd700 1 -- 172.21.15.150:0/4007963010 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f8454010420 con 0x7f8458066920 2023-12-16T20:48:00.231 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.230+0000 7f844effd700 1 --2- 172.21.15.150:0/4007963010 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8444068f60 0x7f844406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:00.232 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.230+0000 7f844effd700 1 -- 172.21.15.150:0/4007963010 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 3364+0+0 (secure 0 0 0) 0x7f8454086550 con 0x7f8458066920 2023-12-16T20:48:00.232 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.230+0000 7f845dc95700 1 --2- 172.21.15.150:0/4007963010 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8444068f60 0x7f844406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:00.232 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.230+0000 7f845dc95700 1 --2- 172.21.15.150:0/4007963010 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8444068f60 0x7f844406b410 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f844800b3b0 tx=0x7f8448009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:00.234 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.234+0000 7f844effd700 1 -- 172.21.15.150:0/4007963010 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8454056ce0 con 0x7f8458066920 2023-12-16T20:48:00.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:59 smithi052 ceph-mon[80761]: Detected new or changed devices on smithi150 2023-12-16T20:48:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:48:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:59 smithi052 ceph-mon[80761]: Adjusting osd_memory_target on smithi150 to 15937M 2023-12-16T20:48:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:48:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:59 smithi052 ceph-mon[80761]: from='osd.3 [v2:172.21.15.150:6800/2861125970,v1:172.21.15.150:6801/2861125970]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T20:48:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:59 smithi052 ceph-mon[80761]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T20:48:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:48:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:00.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:48:00.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:47:59 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:00.426 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:00.425+0000 7f84606fa700 1 -- 172.21.15.150:0/4007963010 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f845810f4c0 con 0x7f8444068f60 2023-12-16T20:48:01.051 INFO:journalctl@ceph.osd.3.smithi150.stdout:Dec 16 20:48:00 smithi150 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-3[93318]: 2023-12-16T20:48:00.844+0000 7f2ccf69c700 -1 osd.3 0 waiting for initial osdmap 2023-12-16T20:48:01.051 INFO:journalctl@ceph.osd.3.smithi150.stdout:Dec 16 20:48:00 smithi150 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-3[93318]: 2023-12-16T20:48:00.851+0000 7f2cca497700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T20:48:01.052 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:00 smithi150 ceph-mon[88000]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-16T20:48:01.052 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:00 smithi150 ceph-mon[88000]: osdmap e23: 4 total, 3 up, 4 in 2023-12-16T20:48:01.052 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T20:48:01.052 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:00 smithi150 ceph-mon[88000]: from='osd.3 [v2:172.21.15.150:6800/2861125970,v1:172.21.15.150:6801/2861125970]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-12-16T20:48:01.052 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:00 smithi150 ceph-mon[88000]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-12-16T20:48:01.052 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:00 smithi150 ceph-mon[88000]: pgmap v62: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-12-16T20:48:01.052 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T20:48:01.052 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T20:48:01.052 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:00 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:01.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:00 smithi052 ceph-mon[80761]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-16T20:48:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:00 smithi052 ceph-mon[80761]: osdmap e23: 4 total, 3 up, 4 in 2023-12-16T20:48:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T20:48:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:00 smithi052 ceph-mon[80761]: from='osd.3 [v2:172.21.15.150:6800/2861125970,v1:172.21.15.150:6801/2861125970]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-12-16T20:48:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:00 smithi052 ceph-mon[80761]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-12-16T20:48:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:00 smithi052 ceph-mon[80761]: pgmap v62: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-12-16T20:48:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T20:48:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T20:48:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:00 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:02.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:01 smithi150 ceph-mon[88000]: from='client.24147 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:48:02.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:01 smithi150 ceph-mon[88000]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-12-16T20:48:02.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:01 smithi150 ceph-mon[88000]: osdmap e24: 4 total, 3 up, 4 in 2023-12-16T20:48:02.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:01 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T20:48:02.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:01 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T20:48:02.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:01 smithi052 ceph-mon[80761]: from='client.24147 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:48:02.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:01 smithi052 ceph-mon[80761]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-12-16T20:48:02.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:01 smithi052 ceph-mon[80761]: osdmap e24: 4 total, 3 up, 4 in 2023-12-16T20:48:02.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:01 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T20:48:02.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:01 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T20:48:03.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:02 smithi150 ceph-mon[88000]: purged_snaps scrub starts 2023-12-16T20:48:03.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:02 smithi150 ceph-mon[88000]: purged_snaps scrub ok 2023-12-16T20:48:03.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:02 smithi150 ceph-mon[88000]: osd.3 [v2:172.21.15.150:6800/2861125970,v1:172.21.15.150:6801/2861125970] boot 2023-12-16T20:48:03.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:02 smithi150 ceph-mon[88000]: osdmap e25: 4 total, 4 up, 4 in 2023-12-16T20:48:03.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:02 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T20:48:03.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:02 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/2224294209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab02c027-f21c-4187-af78-f4daaa844db1"}]: dispatch 2023-12-16T20:48:03.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:02 smithi150 ceph-mon[88000]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab02c027-f21c-4187-af78-f4daaa844db1"}]: dispatch 2023-12-16T20:48:03.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:02 smithi150 ceph-mon[88000]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab02c027-f21c-4187-af78-f4daaa844db1"}]': finished 2023-12-16T20:48:03.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:02 smithi150 ceph-mon[88000]: osdmap e26: 5 total, 4 up, 5 in 2023-12-16T20:48:03.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:02 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:03.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:02 smithi150 ceph-mon[88000]: pgmap v66: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2023-12-16T20:48:03.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:02 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/753155513' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T20:48:03.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:02 smithi052 ceph-mon[80761]: purged_snaps scrub starts 2023-12-16T20:48:03.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:02 smithi052 ceph-mon[80761]: purged_snaps scrub ok 2023-12-16T20:48:03.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:02 smithi052 ceph-mon[80761]: osd.3 [v2:172.21.15.150:6800/2861125970,v1:172.21.15.150:6801/2861125970] boot 2023-12-16T20:48:03.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:02 smithi052 ceph-mon[80761]: osdmap e25: 4 total, 4 up, 4 in 2023-12-16T20:48:03.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:02 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T20:48:03.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:02 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/2224294209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab02c027-f21c-4187-af78-f4daaa844db1"}]: dispatch 2023-12-16T20:48:03.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:02 smithi052 ceph-mon[80761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab02c027-f21c-4187-af78-f4daaa844db1"}]: dispatch 2023-12-16T20:48:03.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:02 smithi052 ceph-mon[80761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab02c027-f21c-4187-af78-f4daaa844db1"}]': finished 2023-12-16T20:48:03.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:02 smithi052 ceph-mon[80761]: osdmap e26: 5 total, 4 up, 5 in 2023-12-16T20:48:03.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:02 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:03.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:02 smithi052 ceph-mon[80761]: pgmap v66: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2023-12-16T20:48:03.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:02 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/753155513' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T20:48:04.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:03 smithi150 ceph-mon[88000]: osdmap e27: 5 total, 4 up, 5 in 2023-12-16T20:48:04.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:03 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:04.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:03 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:48:04.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:03 smithi052 ceph-mon[80761]: osdmap e27: 5 total, 4 up, 5 in 2023-12-16T20:48:04.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:03 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:04.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:03 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:48:05.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:04 smithi150 ceph-mon[88000]: pgmap v68: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-12-16T20:48:05.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:04 smithi150 ceph-mon[88000]: osdmap e28: 5 total, 4 up, 5 in 2023-12-16T20:48:05.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:04 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:05.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:04 smithi052 ceph-mon[80761]: pgmap v68: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-12-16T20:48:05.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:04 smithi052 ceph-mon[80761]: osdmap e28: 5 total, 4 up, 5 in 2023-12-16T20:48:05.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:04 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:06.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:05 smithi150 ceph-mon[88000]: osdmap e29: 5 total, 4 up, 5 in 2023-12-16T20:48:06.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:05 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:06.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:05 smithi052 ceph-mon[80761]: osdmap e29: 5 total, 4 up, 5 in 2023-12-16T20:48:06.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:05 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:06.992 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:06 smithi150 ceph-mon[88000]: pgmap v71: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 143 KiB/s, 0 objects/s recovering 2023-12-16T20:48:07.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:06 smithi052 ceph-mon[80761]: pgmap v71: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 143 KiB/s, 0 objects/s recovering 2023-12-16T20:48:08.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:08 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-16T20:48:08.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:08 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:08.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:08 smithi150 ceph-mon[88000]: Deploying daemon osd.4 on smithi150 2023-12-16T20:48:08.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:08 smithi150 ceph-mon[88000]: pgmap v72: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-12-16T20:48:08.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:08 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-16T20:48:08.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:08 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:08.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:08 smithi052 ceph-mon[80761]: Deploying daemon osd.4 on smithi150 2023-12-16T20:48:08.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:08 smithi052 ceph-mon[80761]: pgmap v72: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-12-16T20:48:11.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:10 smithi150 ceph-mon[88000]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2023-12-16T20:48:11.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:10 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:11.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:10 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:11.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:10 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:48:11.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:10 smithi052 ceph-mon[80761]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2023-12-16T20:48:11.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:10 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:11.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:10 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:11.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:10 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:48:12.526 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:12.525+0000 7f844effd700 1 -- 172.21.15.150:0/4007963010 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f845810f4c0 con 0x7f8444068f60 2023-12-16T20:48:12.526 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 4 on host 'smithi150' 2023-12-16T20:48:12.528 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:12.527+0000 7f84606fa700 1 -- 172.21.15.150:0/4007963010 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8444068f60 msgr2=0x7f844406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:12.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:12.527+0000 7f84606fa700 1 --2- 172.21.15.150:0/4007963010 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8444068f60 0x7f844406b410 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f844800b3b0 tx=0x7f8448009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:12.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:12.527+0000 7f84606fa700 1 -- 172.21.15.150:0/4007963010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8458066920 msgr2=0x7f845810d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:12.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:12.527+0000 7f84606fa700 1 --2- 172.21.15.150:0/4007963010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8458066920 0x7f845810d550 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f845400b790 tx=0x7f8454009250 comp rx=0 tx=0).stop 2023-12-16T20:48:12.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:12.527+0000 7f84606fa700 1 -- 172.21.15.150:0/4007963010 shutdown_connections 2023-12-16T20:48:12.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:12.527+0000 7f84606fa700 1 --2- 172.21.15.150:0/4007963010 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8444068f60 0x7f844406b410 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:12.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:12.527+0000 7f84606fa700 1 --2- 172.21.15.150:0/4007963010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8458066920 0x7f845810d550 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:12.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:12.527+0000 7f84606fa700 1 --2- 172.21.15.150:0/4007963010 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84580ffd70 0x7f845810da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:12.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:12.527+0000 7f84606fa700 1 -- 172.21.15.150:0/4007963010 >> 172.21.15.150:0/4007963010 conn(0x7f84580f71d0 msgr2=0x7f84580f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:12.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:12.527+0000 7f84606fa700 1 -- 172.21.15.150:0/4007963010 shutdown_connections 2023-12-16T20:48:12.529 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:12.527+0000 7f84606fa700 1 -- 172.21.15.150:0/4007963010 wait complete. 2023-12-16T20:48:12.818 DEBUG:teuthology.orchestra.run.smithi150:osd.4> sudo journalctl -f -n 0 -u ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@osd.4.service 2023-12-16T20:48:12.820 INFO:tasks.cephadm:Deploying osd.5 on smithi150 with /dev/vg_nvme/lv_2... 2023-12-16T20:48:12.820 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-12-16T20:48:13.056 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:12 smithi150 ceph-mon[88000]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-16T20:48:13.056 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:12 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:13.056 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:12 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:13.056 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:12 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:13.056 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:12 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:13.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:12 smithi052 ceph-mon[80761]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-16T20:48:13.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:12 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:13.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:12 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:13.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:12 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:13.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:12 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:13.633 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi150/config 2023-12-16T20:48:14.550 INFO:journalctl@ceph.osd.4.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-4[100174]: 2023-12-16T20:48:14.453+0000 7fdbf8b7f540 -1 osd.4 0 log_to_monitors true 2023-12-16T20:48:14.658 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-12-16T20:48:14.658 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-12-16T20:48:14.659 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2023-12-16T20:48:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2023-12-16T20:48:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012881 s, 814 MB/s 2023-12-16T20:48:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2023-12-16T20:48:14.949 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_2 2023-12-16T20:48:15.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-mon[88000]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-16T20:48:15.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-mon[88000]: from='osd.4 [v2:172.21.15.150:6808/2997946868,v1:172.21.15.150:6809/2997946868]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T20:48:15.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-mon[88000]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T20:48:15.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:15.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:15.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:48:15.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:48:15.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:15.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:15.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:48:15.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:15.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:48:15.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:15.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:48:15.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:14 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:15.200 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi150/config 2023-12-16T20:48:15.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:14 smithi052 ceph-mon[80761]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-16T20:48:15.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:14 smithi052 ceph-mon[80761]: from='osd.4 [v2:172.21.15.150:6808/2997946868,v1:172.21.15.150:6809/2997946868]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T20:48:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:14 smithi052 ceph-mon[80761]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T20:48:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:14 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:14 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:14 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:48:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:14 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:48:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:14 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:14 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:14 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:48:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:14 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:14 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:48:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:14 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:14 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:48:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:14 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:15.902 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.900+0000 7fb822a64700 1 -- 172.21.15.150:0/936909853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb81c0fdf60 msgr2=0x7fb81c0fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:15.902 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.900+0000 7fb822a64700 1 --2- 172.21.15.150:0/936909853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb81c0fdf60 0x7fb81c0fe3b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb808004940 tx=0x7fb808009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:15.902 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.900+0000 7fb822a64700 1 -- 172.21.15.150:0/936909853 shutdown_connections 2023-12-16T20:48:15.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.900+0000 7fb822a64700 1 --2- 172.21.15.150:0/936909853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb81c0fdf60 0x7fb81c0fe3b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:15.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.900+0000 7fb822a64700 1 --2- 172.21.15.150:0/936909853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb81c0faed0 0x7fb81c0fb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:15.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.900+0000 7fb822a64700 1 -- 172.21.15.150:0/936909853 >> 172.21.15.150:0/936909853 conn(0x7fb81c0f6950 msgr2=0x7fb81c0f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:15.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.900+0000 7fb822a64700 1 -- 172.21.15.150:0/936909853 shutdown_connections 2023-12-16T20:48:15.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.900+0000 7fb822a64700 1 -- 172.21.15.150:0/936909853 wait complete. 2023-12-16T20:48:15.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.900+0000 7fb822a64700 1 Processor -- start 2023-12-16T20:48:15.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.900+0000 7fb822a64700 1 -- start start 2023-12-16T20:48:15.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.901+0000 7fb822a64700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb81c0faed0 0x7fb81c10d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:15.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.901+0000 7fb822a64700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb81c0fdf60 0x7fb81c10db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:15.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.901+0000 7fb822a64700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c10e100 con 0x7fb81c0faed0 2023-12-16T20:48:15.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.901+0000 7fb822a64700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c10e9a0 con 0x7fb81c0fdf60 2023-12-16T20:48:15.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.901+0000 7fb81bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb81c0fdf60 0x7fb81c10db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:15.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.901+0000 7fb81bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb81c0fdf60 0x7fb81c10db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47006/0 (socket says 172.21.15.150:47006) 2023-12-16T20:48:15.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.901+0000 7fb81bfff700 1 -- 172.21.15.150:0/3887233791 learned_addr learned my addr 172.21.15.150:0/3887233791 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:48:15.903 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.901+0000 7fb820800700 1 --2- 172.21.15.150:0/3887233791 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb81c0faed0 0x7fb81c10d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:15.904 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.901+0000 7fb81bfff700 1 -- 172.21.15.150:0/3887233791 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb81c0faed0 msgr2=0x7fb81c10d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:15.904 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.901+0000 7fb81bfff700 1 --2- 172.21.15.150:0/3887233791 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb81c0faed0 0x7fb81c10d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:15.904 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.901+0000 7fb81bfff700 1 -- 172.21.15.150:0/3887233791 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb808005040 con 0x7fb81c0fdf60 2023-12-16T20:48:15.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.901+0000 7fb81bfff700 1 --2- 172.21.15.150:0/3887233791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb81c0fdf60 0x7fb81c10db00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb80800b3b0 tx=0x7fb808009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:15.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.902+0000 7fb8197fa700 1 -- 172.21.15.150:0/3887233791 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb80800e040 con 0x7fb81c0fdf60 2023-12-16T20:48:15.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.902+0000 7fb822a64700 1 -- 172.21.15.150:0/3887233791 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81c10ec20 con 0x7fb81c0fdf60 2023-12-16T20:48:15.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.902+0000 7fb8197fa700 1 -- 172.21.15.150:0/3887233791 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb808013420 con 0x7fb81c0fdf60 2023-12-16T20:48:15.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.902+0000 7fb8197fa700 1 -- 172.21.15.150:0/3887233791 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb80801c650 con 0x7fb81c0fdf60 2023-12-16T20:48:15.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.902+0000 7fb822a64700 1 -- 172.21.15.150:0/3887233791 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81c10f160 con 0x7fb81c0fdf60 2023-12-16T20:48:15.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.903+0000 7fb822a64700 1 -- 172.21.15.150:0/3887233791 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81c1075f0 con 0x7fb81c0fdf60 2023-12-16T20:48:15.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.903+0000 7fb8197fa700 1 -- 172.21.15.150:0/3887233791 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb808013590 con 0x7fb81c0fdf60 2023-12-16T20:48:15.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.903+0000 7fb8197fa700 1 --2- 172.21.15.150:0/3887233791 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb80c068f60 0x7fb80c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:15.905 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.904+0000 7fb8197fa700 1 -- 172.21.15.150:0/3887233791 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3906+0+0 (secure 0 0 0) 0x7fb808087b00 con 0x7fb81c0fdf60 2023-12-16T20:48:15.906 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.904+0000 7fb820800700 1 --2- 172.21.15.150:0/3887233791 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb80c068f60 0x7fb80c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:15.906 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.904+0000 7fb820800700 1 --2- 172.21.15.150:0/3887233791 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb80c068f60 0x7fb80c06b410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb81c065960 tx=0x7fb810009040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:15.909 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:15.908+0000 7fb8197fa700 1 -- 172.21.15.150:0/3887233791 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb808058120 con 0x7fb81c0fdf60 2023-12-16T20:48:16.111 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:16.109+0000 7fb822a64700 1 -- 172.21.15.150:0/3887233791 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb81c02b1f0 con 0x7fb80c068f60 2023-12-16T20:48:16.188 INFO:journalctl@ceph.osd.4.smithi150.stdout:Dec 16 20:48:15 smithi150 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-4[100174]: 2023-12-16T20:48:15.976+0000 7fdbef364700 -1 osd.4 0 waiting for initial osdmap 2023-12-16T20:48:16.188 INFO:journalctl@ceph.osd.4.smithi150.stdout:Dec 16 20:48:15 smithi150 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-4[100174]: 2023-12-16T20:48:15.982+0000 7fdbea15f700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T20:48:16.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:15 smithi150 ceph-mon[88000]: Detected new or changed devices on smithi150 2023-12-16T20:48:16.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:15 smithi150 ceph-mon[88000]: Adjusting osd_memory_target on smithi150 to 7968M 2023-12-16T20:48:16.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:15 smithi150 ceph-mon[88000]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-16T20:48:16.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:15 smithi150 ceph-mon[88000]: osdmap e30: 5 total, 4 up, 5 in 2023-12-16T20:48:16.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:15 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:16.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:15 smithi150 ceph-mon[88000]: from='osd.4 [v2:172.21.15.150:6808/2997946868,v1:172.21.15.150:6809/2997946868]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-12-16T20:48:16.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:15 smithi150 ceph-mon[88000]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-12-16T20:48:16.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:15 smithi052 ceph-mon[80761]: Detected new or changed devices on smithi150 2023-12-16T20:48:16.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:15 smithi052 ceph-mon[80761]: Adjusting osd_memory_target on smithi150 to 7968M 2023-12-16T20:48:16.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:15 smithi052 ceph-mon[80761]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-16T20:48:16.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:15 smithi052 ceph-mon[80761]: osdmap e30: 5 total, 4 up, 5 in 2023-12-16T20:48:16.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:15 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:16.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:15 smithi052 ceph-mon[80761]: from='osd.4 [v2:172.21.15.150:6808/2997946868,v1:172.21.15.150:6809/2997946868]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-12-16T20:48:16.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:15 smithi052 ceph-mon[80761]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-12-16T20:48:17.057 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:16 smithi150 ceph-mon[88000]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-16T20:48:17.057 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:16 smithi150 ceph-mon[88000]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-12-16T20:48:17.057 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:16 smithi150 ceph-mon[88000]: osdmap e31: 5 total, 4 up, 5 in 2023-12-16T20:48:17.058 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:16 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:17.058 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:16 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:17.058 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:16 smithi150 ceph-mon[88000]: from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:48:17.058 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:16 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T20:48:17.058 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:16 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T20:48:17.058 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:16 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:17.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:16 smithi052 ceph-mon[80761]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-16T20:48:17.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:16 smithi052 ceph-mon[80761]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-12-16T20:48:17.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:16 smithi052 ceph-mon[80761]: osdmap e31: 5 total, 4 up, 5 in 2023-12-16T20:48:17.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:16 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:17.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:16 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:17.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:16 smithi052 ceph-mon[80761]: from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:48:17.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:16 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T20:48:17.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:16 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T20:48:17.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:16 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:18.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:17 smithi052 ceph-mon[80761]: purged_snaps scrub starts 2023-12-16T20:48:18.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:17 smithi052 ceph-mon[80761]: purged_snaps scrub ok 2023-12-16T20:48:18.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:17 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:18.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:17 smithi052 ceph-mon[80761]: osd.4 [v2:172.21.15.150:6808/2997946868,v1:172.21.15.150:6809/2997946868] boot 2023-12-16T20:48:18.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:17 smithi052 ceph-mon[80761]: osdmap e32: 5 total, 5 up, 5 in 2023-12-16T20:48:18.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:17 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:18.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:17 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/1758393233' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f62123bf-fcd1-4253-b8c7-78c2379bb90b"}]: dispatch 2023-12-16T20:48:18.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:17 smithi052 ceph-mon[80761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f62123bf-fcd1-4253-b8c7-78c2379bb90b"}]: dispatch 2023-12-16T20:48:18.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:17 smithi052 ceph-mon[80761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f62123bf-fcd1-4253-b8c7-78c2379bb90b"}]': finished 2023-12-16T20:48:18.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:17 smithi052 ceph-mon[80761]: osdmap e33: 6 total, 5 up, 6 in 2023-12-16T20:48:18.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:17 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T20:48:18.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:17 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:48:18.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:17 smithi150 ceph-mon[88000]: purged_snaps scrub starts 2023-12-16T20:48:18.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:17 smithi150 ceph-mon[88000]: purged_snaps scrub ok 2023-12-16T20:48:18.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:17 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:18.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:17 smithi150 ceph-mon[88000]: osd.4 [v2:172.21.15.150:6808/2997946868,v1:172.21.15.150:6809/2997946868] boot 2023-12-16T20:48:18.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:17 smithi150 ceph-mon[88000]: osdmap e32: 5 total, 5 up, 5 in 2023-12-16T20:48:18.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:17 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T20:48:18.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:17 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/1758393233' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f62123bf-fcd1-4253-b8c7-78c2379bb90b"}]: dispatch 2023-12-16T20:48:18.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:17 smithi150 ceph-mon[88000]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f62123bf-fcd1-4253-b8c7-78c2379bb90b"}]: dispatch 2023-12-16T20:48:18.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:17 smithi150 ceph-mon[88000]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f62123bf-fcd1-4253-b8c7-78c2379bb90b"}]': finished 2023-12-16T20:48:18.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:17 smithi150 ceph-mon[88000]: osdmap e33: 6 total, 5 up, 6 in 2023-12-16T20:48:18.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:17 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T20:48:18.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:17 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:48:19.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:18 smithi052 ceph-mon[80761]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-16T20:48:19.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:18 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/2960109026' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T20:48:19.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:18 smithi150 ceph-mon[88000]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-16T20:48:19.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:18 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/2960109026' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T20:48:21.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:20 smithi052 ceph-mon[80761]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:21.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:20 smithi150 ceph-mon[88000]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:23.187 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:22 smithi150 ceph-mon[88000]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:23.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:22 smithi052 ceph-mon[80761]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:24.048 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:23 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-16T20:48:24.049 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:23 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:24.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-16T20:48:24.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:25.107 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:24 smithi150 ceph-mon[88000]: Deploying daemon osd.5 on smithi150 2023-12-16T20:48:25.107 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:24 smithi150 ceph-mon[88000]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:25.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:24 smithi052 ceph-mon[80761]: Deploying daemon osd.5 on smithi150 2023-12-16T20:48:25.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:24 smithi052 ceph-mon[80761]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:27.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:26 smithi150 ceph-mon[88000]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:27.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:26 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:27.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:26 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:27.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:26 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:48:27.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:26 smithi052 ceph-mon[80761]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:27.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:27.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:27.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:48:28.112 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 5 on host 'smithi150' 2023-12-16T20:48:28.112 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:28.110+0000 7fb8197fa700 1 -- 172.21.15.150:0/3887233791 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb81c02b1f0 con 0x7fb80c068f60 2023-12-16T20:48:28.113 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:28.112+0000 7fb822a64700 1 -- 172.21.15.150:0/3887233791 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb80c068f60 msgr2=0x7fb80c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:28.113 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:28.112+0000 7fb822a64700 1 --2- 172.21.15.150:0/3887233791 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb80c068f60 0x7fb80c06b410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb81c065960 tx=0x7fb810009040 comp rx=0 tx=0).stop 2023-12-16T20:48:28.114 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:28.112+0000 7fb822a64700 1 -- 172.21.15.150:0/3887233791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb81c0fdf60 msgr2=0x7fb81c10db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:28.114 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:28.112+0000 7fb822a64700 1 --2- 172.21.15.150:0/3887233791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb81c0fdf60 0x7fb81c10db00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb80800b3b0 tx=0x7fb808009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:28.114 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:28.112+0000 7fb822a64700 1 -- 172.21.15.150:0/3887233791 shutdown_connections 2023-12-16T20:48:28.114 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:28.112+0000 7fb822a64700 1 --2- 172.21.15.150:0/3887233791 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb80c068f60 0x7fb80c06b410 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:28.114 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:28.112+0000 7fb822a64700 1 --2- 172.21.15.150:0/3887233791 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb81c0faed0 0x7fb81c10d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:28.114 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:28.112+0000 7fb822a64700 1 --2- 172.21.15.150:0/3887233791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb81c0fdf60 0x7fb81c10db00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:28.114 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:28.112+0000 7fb822a64700 1 -- 172.21.15.150:0/3887233791 >> 172.21.15.150:0/3887233791 conn(0x7fb81c0f6950 msgr2=0x7fb81c0f8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:28.114 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:28.112+0000 7fb822a64700 1 -- 172.21.15.150:0/3887233791 shutdown_connections 2023-12-16T20:48:28.114 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:28.112+0000 7fb822a64700 1 -- 172.21.15.150:0/3887233791 wait complete. 2023-12-16T20:48:28.433 DEBUG:teuthology.orchestra.run.smithi150:osd.5> sudo journalctl -f -n 0 -u ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@osd.5.service 2023-12-16T20:48:28.435 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2023-12-16T20:48:28.435 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph osd stat -f json 2023-12-16T20:48:28.721 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:29.394 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:29 smithi052 ceph-mon[80761]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:29.394 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:29 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:29.394 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:29 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:29.394 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:29 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:29.394 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:29 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:29.414 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:29 smithi150 ceph-mon[88000]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:29.414 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:29 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:29.415 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:29 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:29.415 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:29 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:29.415 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:29 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.593+0000 7f485a520700 1 -- 172.21.15.52:0/957219446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48540fee00 msgr2=0x7f48540ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.593+0000 7f485a520700 1 --2- 172.21.15.52:0/957219446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48540fee00 0x7f48540ff210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4844004940 tx=0x7f4844009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.594+0000 7f485a520700 1 -- 172.21.15.52:0/957219446 shutdown_connections 2023-12-16T20:48:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.594+0000 7f485a520700 1 --2- 172.21.15.52:0/957219446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48540fff70 0x7f48541003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.594+0000 7f485a520700 1 --2- 172.21.15.52:0/957219446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48540fee00 0x7f48540ff210 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.594+0000 7f485a520700 1 -- 172.21.15.52:0/957219446 >> 172.21.15.52:0/957219446 conn(0x7f48540fa5d0 msgr2=0x7f48540fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.594+0000 7f485a520700 1 -- 172.21.15.52:0/957219446 shutdown_connections 2023-12-16T20:48:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.594+0000 7f485a520700 1 -- 172.21.15.52:0/957219446 wait complete. 2023-12-16T20:48:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.594+0000 7f485a520700 1 Processor -- start 2023-12-16T20:48:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.594+0000 7f485a520700 1 -- start start 2023-12-16T20:48:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.594+0000 7f485a520700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48540fee00 0x7f485410f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f485a520700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48540fff70 0x7f485410fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f485a520700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4854110200 con 0x7f48540fee00 2023-12-16T20:48:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f485a520700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4854110aa0 con 0x7f48540fff70 2023-12-16T20:48:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f4853fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48540fee00 0x7f485410f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f48537fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48540fff70 0x7f485410fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f4853fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48540fee00 0x7f485410f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37222/0 (socket says 172.21.15.52:37222) 2023-12-16T20:48:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f4853fff700 1 -- 172.21.15.52:0/102482139 learned_addr learned my addr 172.21.15.52:0/102482139 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f4853fff700 1 -- 172.21.15.52:0/102482139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48540fff70 msgr2=0x7f485410fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f4853fff700 1 --2- 172.21.15.52:0/102482139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48540fff70 0x7f485410fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f4853fff700 1 -- 172.21.15.52:0/102482139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4844005040 con 0x7f48540fee00 2023-12-16T20:48:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f48537fe700 1 --2- 172.21.15.52:0/102482139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48540fff70 0x7f485410fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T20:48:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f4853fff700 1 --2- 172.21.15.52:0/102482139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48540fee00 0x7f485410f6c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4844000c00 tx=0x7f4844009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f4850ff9700 1 -- 172.21.15.52:0/102482139 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4844021070 con 0x7f48540fee00 2023-12-16T20:48:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f4850ff9700 1 -- 172.21.15.52:0/102482139 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f484401c400 con 0x7f48540fee00 2023-12-16T20:48:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f4850ff9700 1 -- 172.21.15.52:0/102482139 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f484400e070 con 0x7f48540fee00 2023-12-16T20:48:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.595+0000 7f485a520700 1 -- 172.21.15.52:0/102482139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4854110d20 con 0x7f48540fee00 2023-12-16T20:48:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.596+0000 7f485a520700 1 -- 172.21.15.52:0/102482139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4854111200 con 0x7f48540fee00 2023-12-16T20:48:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.597+0000 7f485a520700 1 -- 172.21.15.52:0/102482139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485404bb80 con 0x7f48540fee00 2023-12-16T20:48:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.597+0000 7f4850ff9700 1 -- 172.21.15.52:0/102482139 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f484401c9c0 con 0x7f48540fee00 2023-12-16T20:48:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.597+0000 7f4850ff9700 1 --2- 172.21.15.52:0/102482139 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f483c069030 0x7f483c06b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.597+0000 7f4850ff9700 1 -- 172.21.15.52:0/102482139 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4317+0+0 (secure 0 0 0) 0x7f4844087af0 con 0x7f48540fee00 2023-12-16T20:48:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.597+0000 7f48537fe700 1 --2- 172.21.15.52:0/102482139 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f483c069030 0x7f483c06b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.598+0000 7f48537fe700 1 --2- 172.21.15.52:0/102482139 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f483c069030 0x7f483c06b4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4848009b80 tx=0x7f484800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:29.602 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.602+0000 7f4850ff9700 1 -- 172.21.15.52:0/102482139 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4844057eb0 con 0x7f48540fee00 2023-12-16T20:48:29.801 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.798+0000 7f485a520700 1 -- 172.21.15.52:0/102482139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f48541077b0 con 0x7f48540fee00 2023-12-16T20:48:29.802 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.799+0000 7f4850ff9700 1 -- 172.21.15.52:0/102482139 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v33) v1 ==== 74+0+130 (secure 0 0 0) 0x7f4844057a40 con 0x7f48540fee00 2023-12-16T20:48:29.802 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:48:29.802 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.801+0000 7f485a520700 1 -- 172.21.15.52:0/102482139 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f483c069030 msgr2=0x7f483c06b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:29.803 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.801+0000 7f485a520700 1 --2- 172.21.15.52:0/102482139 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f483c069030 0x7f483c06b4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4848009b80 tx=0x7f484800b040 comp rx=0 tx=0).stop 2023-12-16T20:48:29.803 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.801+0000 7f485a520700 1 -- 172.21.15.52:0/102482139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48540fee00 msgr2=0x7f485410f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:29.803 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.801+0000 7f485a520700 1 --2- 172.21.15.52:0/102482139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48540fee00 0x7f485410f6c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4844000c00 tx=0x7f4844009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:29.803 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.801+0000 7f485a520700 1 -- 172.21.15.52:0/102482139 shutdown_connections 2023-12-16T20:48:29.803 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.801+0000 7f485a520700 1 --2- 172.21.15.52:0/102482139 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f483c069030 0x7f483c06b4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:29.803 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.801+0000 7f485a520700 1 --2- 172.21.15.52:0/102482139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48540fee00 0x7f485410f6c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:29.803 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.801+0000 7f485a520700 1 --2- 172.21.15.52:0/102482139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48540fff70 0x7f485410fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:29.803 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.801+0000 7f485a520700 1 -- 172.21.15.52:0/102482139 >> 172.21.15.52:0/102482139 conn(0x7f48540fa5d0 msgr2=0x7f4854103230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:29.803 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.801+0000 7f485a520700 1 -- 172.21.15.52:0/102482139 shutdown_connections 2023-12-16T20:48:29.803 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:29.801+0000 7f485a520700 1 -- 172.21.15.52:0/102482139 wait complete. 2023-12-16T20:48:30.145 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":33,"num_osds":6,"num_up_osds":5,"osd_up_since":1702759696,"num_in_osds":6,"osd_in_since":1702759697,"num_remapped_pgs":0} 2023-12-16T20:48:30.382 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:30 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/102482139' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-16T20:48:30.382 INFO:journalctl@ceph.osd.5.smithi150.stdout:Dec 16 20:48:30 smithi150 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-5[106921]: 2023-12-16T20:48:30.159+0000 7f1332f88540 -1 osd.5 0 log_to_monitors true 2023-12-16T20:48:30.403 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:30 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/102482139' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-16T20:48:31.146 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph osd stat -f json 2023-12-16T20:48:31.379 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:31 smithi052 ceph-mon[80761]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:31.380 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:31 smithi052 ceph-mon[80761]: from='osd.5 [v2:172.21.15.150:6816/4046379778,v1:172.21.15.150:6817/4046379778]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T20:48:31.380 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:31 smithi052 ceph-mon[80761]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T20:48:31.380 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:31.380 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:31.380 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:48:31.380 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:48:31.380 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:48:31.380 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:31.380 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:31.380 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:48:31.380 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:31.380 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:31 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:48:31.415 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:31 smithi150 ceph-mon[88000]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:31 smithi150 ceph-mon[88000]: from='osd.5 [v2:172.21.15.150:6816/4046379778,v1:172.21.15.150:6817/4046379778]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T20:48:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:31 smithi150 ceph-mon[88000]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T20:48:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:31 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:31 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:31 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:48:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:31 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:48:31.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:31 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T20:48:31.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:31 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:31.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:31 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:31.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:31 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:48:31.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:31 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:31.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:31 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:48:32.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:32 smithi052 ceph-mon[80761]: Detected new or changed devices on smithi150 2023-12-16T20:48:32.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:32 smithi052 ceph-mon[80761]: Adjusting osd_memory_target on smithi150 to 5312M 2023-12-16T20:48:32.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:32 smithi052 ceph-mon[80761]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-16T20:48:32.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:32 smithi052 ceph-mon[80761]: osdmap e34: 6 total, 5 up, 6 in 2023-12-16T20:48:32.295 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:32 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T20:48:32.295 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:32 smithi052 ceph-mon[80761]: from='osd.5 [v2:172.21.15.150:6816/4046379778,v1:172.21.15.150:6817/4046379778]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-12-16T20:48:32.295 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:32 smithi052 ceph-mon[80761]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-12-16T20:48:32.295 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:32 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:32.295 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:32 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:48:32.295 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:32 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:32.395 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.393+0000 7f7c35d85700 1 -- 172.21.15.52:0/608670269 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c300fbaf0 msgr2=0x7f7c300fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:32.396 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.393+0000 7f7c35d85700 1 --2- 172.21.15.52:0/608670269 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c300fbaf0 0x7f7c300fbf00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7c20004940 tx=0x7f7c20009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:32.396 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.393+0000 7f7c35d85700 1 -- 172.21.15.52:0/608670269 shutdown_connections 2023-12-16T20:48:32.396 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.393+0000 7f7c35d85700 1 --2- 172.21.15.52:0/608670269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c300fcd00 0x7f7c300fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:32.396 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.393+0000 7f7c35d85700 1 --2- 172.21.15.52:0/608670269 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c300fbaf0 0x7f7c300fbf00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:32.396 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.393+0000 7f7c35d85700 1 -- 172.21.15.52:0/608670269 >> 172.21.15.52:0/608670269 conn(0x7f7c300f7190 msgr2=0x7f7c300f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:32.396 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.393+0000 7f7c35d85700 1 -- 172.21.15.52:0/608670269 shutdown_connections 2023-12-16T20:48:32.396 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.393+0000 7f7c35d85700 1 -- 172.21.15.52:0/608670269 wait complete. 2023-12-16T20:48:32.396 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.393+0000 7f7c35d85700 1 Processor -- start 2023-12-16T20:48:32.396 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.394+0000 7f7c35d85700 1 -- start start 2023-12-16T20:48:32.396 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.394+0000 7f7c35d85700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c300fbaf0 0x7f7c3010c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:32.396 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.394+0000 7f7c35d85700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c300fcd00 0x7f7c3010ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:32.396 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.394+0000 7f7c35d85700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3010d0b0 con 0x7f7c300fcd00 2023-12-16T20:48:32.396 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.394+0000 7f7c35d85700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3010d8f0 con 0x7f7c300fbaf0 2023-12-16T20:48:32.397 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.394+0000 7f7c2effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c300fcd00 0x7f7c3010ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:32.397 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.394+0000 7f7c2effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c300fcd00 0x7f7c3010ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37244/0 (socket says 172.21.15.52:37244) 2023-12-16T20:48:32.397 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.394+0000 7f7c2effd700 1 -- 172.21.15.52:0/1971034897 learned_addr learned my addr 172.21.15.52:0/1971034897 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:32.397 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.394+0000 7f7c2f7fe700 1 --2- 172.21.15.52:0/1971034897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c300fbaf0 0x7f7c3010c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:32.397 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.394+0000 7f7c2effd700 1 -- 172.21.15.52:0/1971034897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c300fbaf0 msgr2=0x7f7c3010c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:32.397 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.394+0000 7f7c2effd700 1 --2- 172.21.15.52:0/1971034897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c300fbaf0 0x7f7c3010c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:32.397 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.394+0000 7f7c2effd700 1 -- 172.21.15.52:0/1971034897 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c20005040 con 0x7f7c300fcd00 2023-12-16T20:48:32.397 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.394+0000 7f7c2f7fe700 1 --2- 172.21.15.52:0/1971034897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c300fbaf0 0x7f7c3010c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T20:48:32.397 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.395+0000 7f7c2effd700 1 --2- 172.21.15.52:0/1971034897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c300fcd00 0x7f7c3010ca20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400b790 tx=0x7f7c24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:32.397 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.395+0000 7f7c17fff700 1 -- 172.21.15.52:0/1971034897 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c2400f770 con 0x7f7c300fcd00 2023-12-16T20:48:32.397 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.395+0000 7f7c35d85700 1 -- 172.21.15.52:0/1971034897 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3010dbd0 con 0x7f7c300fcd00 2023-12-16T20:48:32.397 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.395+0000 7f7c17fff700 1 -- 172.21.15.52:0/1971034897 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c2400fd80 con 0x7f7c300fcd00 2023-12-16T20:48:32.397 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.395+0000 7f7c17fff700 1 -- 172.21.15.52:0/1971034897 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c2400d310 con 0x7f7c300fcd00 2023-12-16T20:48:32.398 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.395+0000 7f7c35d85700 1 -- 172.21.15.52:0/1971034897 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3010e110 con 0x7f7c300fcd00 2023-12-16T20:48:32.398 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.396+0000 7f7c35d85700 1 -- 172.21.15.52:0/1971034897 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3004bb80 con 0x7f7c300fcd00 2023-12-16T20:48:32.398 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.396+0000 7f7c17fff700 1 -- 172.21.15.52:0/1971034897 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f7c2400f8d0 con 0x7f7c300fcd00 2023-12-16T20:48:32.398 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.396+0000 7f7c17fff700 1 --2- 172.21.15.52:0/1971034897 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7c18068f60 0x7f7c1806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:32.399 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.397+0000 7f7c17fff700 1 -- 172.21.15.52:0/1971034897 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4354+0+0 (secure 0 0 0) 0x7f7c24086850 con 0x7f7c300fcd00 2023-12-16T20:48:32.399 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.397+0000 7f7c2f7fe700 1 --2- 172.21.15.52:0/1971034897 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7c18068f60 0x7f7c1806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:32.399 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.397+0000 7f7c2f7fe700 1 --2- 172.21.15.52:0/1971034897 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7c18068f60 0x7f7c1806b410 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7c20009fd0 tx=0x7f7c20009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:32.401 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.401+0000 7f7c17fff700 1 -- 172.21.15.52:0/1971034897 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7c24056ca0 con 0x7f7c300fcd00 2023-12-16T20:48:32.438 INFO:journalctl@ceph.osd.5.smithi150.stdout:Dec 16 20:48:32 smithi150 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-5[106921]: 2023-12-16T20:48:32.108+0000 7f132976d700 -1 osd.5 0 waiting for initial osdmap 2023-12-16T20:48:32.438 INFO:journalctl@ceph.osd.5.smithi150.stdout:Dec 16 20:48:32 smithi150 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-5[106921]: 2023-12-16T20:48:32.118+0000 7f1324d69700 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T20:48:32.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:32 smithi150 ceph-mon[88000]: Detected new or changed devices on smithi150 2023-12-16T20:48:32.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:32 smithi150 ceph-mon[88000]: Adjusting osd_memory_target on smithi150 to 5312M 2023-12-16T20:48:32.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:32 smithi150 ceph-mon[88000]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-16T20:48:32.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:32 smithi150 ceph-mon[88000]: osdmap e34: 6 total, 5 up, 6 in 2023-12-16T20:48:32.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T20:48:32.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:32 smithi150 ceph-mon[88000]: from='osd.5 [v2:172.21.15.150:6816/4046379778,v1:172.21.15.150:6817/4046379778]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-12-16T20:48:32.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:32 smithi150 ceph-mon[88000]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-12-16T20:48:32.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:48:32.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:48:32.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:48:32.602 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.601+0000 7f7c35d85700 1 -- 172.21.15.52:0/1971034897 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f7c3010e560 con 0x7f7c300fcd00 2023-12-16T20:48:32.602 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.601+0000 7f7c17fff700 1 -- 172.21.15.52:0/1971034897 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v35) v1 ==== 74+0+130 (secure 0 0 0) 0x7f7c2405ae60 con 0x7f7c300fcd00 2023-12-16T20:48:32.602 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:48:32.604 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.603+0000 7f7c35d85700 1 -- 172.21.15.52:0/1971034897 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7c18068f60 msgr2=0x7f7c1806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:32.604 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.603+0000 7f7c35d85700 1 --2- 172.21.15.52:0/1971034897 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7c18068f60 0x7f7c1806b410 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7c20009fd0 tx=0x7f7c20009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:32.604 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.603+0000 7f7c35d85700 1 -- 172.21.15.52:0/1971034897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c300fcd00 msgr2=0x7f7c3010ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:32.604 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.603+0000 7f7c35d85700 1 --2- 172.21.15.52:0/1971034897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c300fcd00 0x7f7c3010ca20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400b790 tx=0x7f7c24009250 comp rx=0 tx=0).stop 2023-12-16T20:48:32.605 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.603+0000 7f7c35d85700 1 -- 172.21.15.52:0/1971034897 shutdown_connections 2023-12-16T20:48:32.605 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.603+0000 7f7c35d85700 1 --2- 172.21.15.52:0/1971034897 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7c18068f60 0x7f7c1806b410 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:32.605 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.603+0000 7f7c35d85700 1 --2- 172.21.15.52:0/1971034897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c300fbaf0 0x7f7c3010c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:32.605 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.603+0000 7f7c35d85700 1 --2- 172.21.15.52:0/1971034897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c300fcd00 0x7f7c3010ca20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:32.605 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.603+0000 7f7c35d85700 1 -- 172.21.15.52:0/1971034897 >> 172.21.15.52:0/1971034897 conn(0x7f7c300f7190 msgr2=0x7f7c300fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:32.605 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.604+0000 7f7c35d85700 1 -- 172.21.15.52:0/1971034897 shutdown_connections 2023-12-16T20:48:32.605 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:32.604+0000 7f7c35d85700 1 -- 172.21.15.52:0/1971034897 wait complete. 2023-12-16T20:48:33.061 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":35,"num_osds":6,"num_up_osds":5,"osd_up_since":1702759696,"num_in_osds":6,"osd_in_since":1702759697,"num_remapped_pgs":0} 2023-12-16T20:48:33.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:33 smithi052 ceph-mon[80761]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:33.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:33 smithi052 ceph-mon[80761]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-12-16T20:48:33.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:33 smithi052 ceph-mon[80761]: osdmap e35: 6 total, 5 up, 6 in 2023-12-16T20:48:33.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:33 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T20:48:33.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:33 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T20:48:33.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:33 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/1971034897' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-16T20:48:33.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:33 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:48:33.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:33 smithi150 ceph-mon[88000]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:33.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:33 smithi150 ceph-mon[88000]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-12-16T20:48:33.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:33 smithi150 ceph-mon[88000]: osdmap e35: 6 total, 5 up, 6 in 2023-12-16T20:48:33.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:33 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T20:48:33.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:33 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T20:48:33.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:33 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/1971034897' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-16T20:48:33.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:33 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:48:34.062 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph osd stat -f json 2023-12-16T20:48:34.344 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:34.435 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:34 smithi052 ceph-mon[80761]: purged_snaps scrub starts 2023-12-16T20:48:34.435 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:34 smithi052 ceph-mon[80761]: purged_snaps scrub ok 2023-12-16T20:48:34.435 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:34 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T20:48:34.435 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:34 smithi052 ceph-mon[80761]: osd.5 [v2:172.21.15.150:6816/4046379778,v1:172.21.15.150:6817/4046379778] boot 2023-12-16T20:48:34.435 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:34 smithi052 ceph-mon[80761]: osdmap e36: 6 total, 6 up, 6 in 2023-12-16T20:48:34.435 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:34 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T20:48:34.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:34 smithi150 ceph-mon[88000]: purged_snaps scrub starts 2023-12-16T20:48:34.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:34 smithi150 ceph-mon[88000]: purged_snaps scrub ok 2023-12-16T20:48:34.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:34 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T20:48:34.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:34 smithi150 ceph-mon[88000]: osd.5 [v2:172.21.15.150:6816/4046379778,v1:172.21.15.150:6817/4046379778] boot 2023-12-16T20:48:34.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:34 smithi150 ceph-mon[88000]: osdmap e36: 6 total, 6 up, 6 in 2023-12-16T20:48:34.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:34 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T20:48:35.270 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:35 smithi052 ceph-mon[80761]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:35.371 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.369+0000 7f821860b700 1 -- 172.21.15.52:0/181527518 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82100fcbf0 msgr2=0x7f82100fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:35.371 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.369+0000 7f821860b700 1 --2- 172.21.15.52:0/181527518 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82100fcbf0 0x7f82100fd000 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8200009ce0 tx=0x7f8200004d10 comp rx=0 tx=0).stop 2023-12-16T20:48:35.372 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.369+0000 7f821860b700 1 -- 172.21.15.52:0/181527518 shutdown_connections 2023-12-16T20:48:35.372 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.369+0000 7f821860b700 1 --2- 172.21.15.52:0/181527518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82100fde00 0x7f82100fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:35.372 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.369+0000 7f821860b700 1 --2- 172.21.15.52:0/181527518 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82100fcbf0 0x7f82100fd000 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:35.372 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.369+0000 7f821860b700 1 -- 172.21.15.52:0/181527518 >> 172.21.15.52:0/181527518 conn(0x7f82100f82d0 msgr2=0x7f82100fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:35.372 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.369+0000 7f821860b700 1 -- 172.21.15.52:0/181527518 shutdown_connections 2023-12-16T20:48:35.372 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.369+0000 7f821860b700 1 -- 172.21.15.52:0/181527518 wait complete. 2023-12-16T20:48:35.372 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.370+0000 7f821860b700 1 Processor -- start 2023-12-16T20:48:35.372 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.370+0000 7f821860b700 1 -- start start 2023-12-16T20:48:35.372 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.370+0000 7f821860b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82100fcbf0 0x7f821010d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:35.372 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.370+0000 7f821860b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82100fde00 0x7f821010dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:35.372 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.370+0000 7f821860b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821010e260 con 0x7f82100fde00 2023-12-16T20:48:35.372 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.370+0000 7f821860b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821010eb00 con 0x7f82100fcbf0 2023-12-16T20:48:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.370+0000 7f8215ba6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82100fde00 0x7f821010dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.370+0000 7f8215ba6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82100fde00 0x7f821010dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42566/0 (socket says 172.21.15.52:42566) 2023-12-16T20:48:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.370+0000 7f8215ba6700 1 -- 172.21.15.52:0/840900624 learned_addr learned my addr 172.21.15.52:0/840900624 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.370+0000 7f82163a7700 1 --2- 172.21.15.52:0/840900624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82100fcbf0 0x7f821010d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.370+0000 7f8215ba6700 1 -- 172.21.15.52:0/840900624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82100fcbf0 msgr2=0x7f821010d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.370+0000 7f8215ba6700 1 --2- 172.21.15.52:0/840900624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82100fcbf0 0x7f821010d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.370+0000 7f8215ba6700 1 -- 172.21.15.52:0/840900624 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8200009940 con 0x7f82100fde00 2023-12-16T20:48:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.370+0000 7f8215ba6700 1 --2- 172.21.15.52:0/840900624 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82100fde00 0x7f821010dc60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8204002a50 tx=0x7f82040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.371+0000 7f820effd700 1 -- 172.21.15.52:0/840900624 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8204007cc0 con 0x7f82100fde00 2023-12-16T20:48:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.371+0000 7f820effd700 1 -- 172.21.15.52:0/840900624 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f820400f040 con 0x7f82100fde00 2023-12-16T20:48:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.371+0000 7f821860b700 1 -- 172.21.15.52:0/840900624 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821010ede0 con 0x7f82100fde00 2023-12-16T20:48:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.371+0000 7f820effd700 1 -- 172.21.15.52:0/840900624 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f820400d9c0 con 0x7f82100fde00 2023-12-16T20:48:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.371+0000 7f821860b700 1 -- 172.21.15.52:0/840900624 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821010f320 con 0x7f82100fde00 2023-12-16T20:48:35.374 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.372+0000 7f821860b700 1 -- 172.21.15.52:0/840900624 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f821004bb80 con 0x7f82100fde00 2023-12-16T20:48:35.374 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.372+0000 7f820effd700 1 -- 172.21.15.52:0/840900624 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f820400db20 con 0x7f82100fde00 2023-12-16T20:48:35.374 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.372+0000 7f820effd700 1 --2- 172.21.15.52:0/840900624 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f81f8068f70 0x7f81f806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:35.375 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.373+0000 7f820effd700 1 -- 172.21.15.52:0/840900624 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8204086790 con 0x7f82100fde00 2023-12-16T20:48:35.375 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.373+0000 7f82163a7700 1 --2- 172.21.15.52:0/840900624 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f81f8068f70 0x7f81f806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:35.375 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.373+0000 7f82163a7700 1 --2- 172.21.15.52:0/840900624 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f81f8068f70 0x7f81f806b420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8200009400 tx=0x7f8200004d10 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:35.377 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.377+0000 7f820effd700 1 -- 172.21.15.52:0/840900624 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f82040569b0 con 0x7f82100fde00 2023-12-16T20:48:35.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:35 smithi150 ceph-mon[88000]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T20:48:35.574 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.573+0000 7f821860b700 1 -- 172.21.15.52:0/840900624 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f8210105590 con 0x7f82100fde00 2023-12-16T20:48:35.574 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.574+0000 7f820effd700 1 -- 172.21.15.52:0/840900624 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v37) v1 ==== 74+0+130 (secure 0 0 0) 0x7f8204056540 con 0x7f82100fde00 2023-12-16T20:48:35.574 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:48:35.577 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.576+0000 7f821860b700 1 -- 172.21.15.52:0/840900624 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f81f8068f70 msgr2=0x7f81f806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:35.577 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.576+0000 7f821860b700 1 --2- 172.21.15.52:0/840900624 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f81f8068f70 0x7f81f806b420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8200009400 tx=0x7f8200004d10 comp rx=0 tx=0).stop 2023-12-16T20:48:35.577 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.576+0000 7f821860b700 1 -- 172.21.15.52:0/840900624 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82100fde00 msgr2=0x7f821010dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:35.577 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.576+0000 7f821860b700 1 --2- 172.21.15.52:0/840900624 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82100fde00 0x7f821010dc60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8204002a50 tx=0x7f82040046b0 comp rx=0 tx=0).stop 2023-12-16T20:48:35.577 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.576+0000 7f821860b700 1 -- 172.21.15.52:0/840900624 shutdown_connections 2023-12-16T20:48:35.577 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.576+0000 7f821860b700 1 --2- 172.21.15.52:0/840900624 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f81f8068f70 0x7f81f806b420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:35.577 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.576+0000 7f821860b700 1 --2- 172.21.15.52:0/840900624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82100fcbf0 0x7f821010d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:35.577 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.576+0000 7f821860b700 1 --2- 172.21.15.52:0/840900624 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82100fde00 0x7f821010dc60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:35.577 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.576+0000 7f821860b700 1 -- 172.21.15.52:0/840900624 >> 172.21.15.52:0/840900624 conn(0x7f82100f82d0 msgr2=0x7f8210101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:35.577 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.576+0000 7f821860b700 1 -- 172.21.15.52:0/840900624 shutdown_connections 2023-12-16T20:48:35.577 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:35.576+0000 7f821860b700 1 -- 172.21.15.52:0/840900624 wait complete. 2023-12-16T20:48:35.918 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":37,"num_osds":6,"num_up_osds":6,"osd_up_since":1702759713,"num_in_osds":6,"osd_in_since":1702759697,"num_remapped_pgs":0} 2023-12-16T20:48:35.919 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-16T20:48:36.183 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:36.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:36 smithi052 ceph-mon[80761]: osdmap e37: 6 total, 6 up, 6 in 2023-12-16T20:48:36.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:36 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/840900624' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-16T20:48:36.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:36 smithi052 ceph-mon[80761]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:36.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:36 smithi150 ceph-mon[88000]: osdmap e37: 6 total, 6 up, 6 in 2023-12-16T20:48:36.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:36 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/840900624' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-16T20:48:36.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:36 smithi150 ceph-mon[88000]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:37.243 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.242+0000 7fee71d40700 1 -- 172.21.15.52:0/3185799319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee6c0fbb40 msgr2=0x7fee6c0fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:37.243 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.242+0000 7fee71d40700 1 --2- 172.21.15.52:0/3185799319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee6c0fbb40 0x7fee6c0fbfb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fee54004940 tx=0x7fee54009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:37.243 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.242+0000 7fee71d40700 1 -- 172.21.15.52:0/3185799319 shutdown_connections 2023-12-16T20:48:37.243 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.242+0000 7fee71d40700 1 --2- 172.21.15.52:0/3185799319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee6c0fbb40 0x7fee6c0fbfb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:37.244 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.242+0000 7fee71d40700 1 --2- 172.21.15.52:0/3185799319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6c101b00 0x7fee6c101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:37.244 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.242+0000 7fee71d40700 1 -- 172.21.15.52:0/3185799319 >> 172.21.15.52:0/3185799319 conn(0x7fee6c0719b0 msgr2=0x7fee6c071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:37.244 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.242+0000 7fee71d40700 1 -- 172.21.15.52:0/3185799319 shutdown_connections 2023-12-16T20:48:37.244 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.242+0000 7fee71d40700 1 -- 172.21.15.52:0/3185799319 wait complete. 2023-12-16T20:48:37.244 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.243+0000 7fee71d40700 1 Processor -- start 2023-12-16T20:48:37.244 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.243+0000 7fee71d40700 1 -- start start 2023-12-16T20:48:37.248 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.243+0000 7fee71d40700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee6c0fbb40 0x7fee6c10e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:37.248 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.243+0000 7fee71d40700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6c101b00 0x7fee6c10ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:37.248 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.243+0000 7fee71d40700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c10f200 con 0x7fee6c0fbb40 2023-12-16T20:48:37.248 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.243+0000 7fee71d40700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c10f370 con 0x7fee6c101b00 2023-12-16T20:48:37.248 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.243+0000 7fee6b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee6c0fbb40 0x7fee6c10e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:37.248 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.243+0000 7fee6b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee6c0fbb40 0x7fee6c10e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42590/0 (socket says 172.21.15.52:42590) 2023-12-16T20:48:37.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.243+0000 7fee6b7fe700 1 -- 172.21.15.52:0/1980773063 learned_addr learned my addr 172.21.15.52:0/1980773063 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:37.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.243+0000 7fee63fff700 1 --2- 172.21.15.52:0/1980773063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6c101b00 0x7fee6c10ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:37.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.243+0000 7fee6b7fe700 1 -- 172.21.15.52:0/1980773063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6c101b00 msgr2=0x7fee6c10ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:37.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.243+0000 7fee6b7fe700 1 --2- 172.21.15.52:0/1980773063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6c101b00 0x7fee6c10ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:37.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.243+0000 7fee6b7fe700 1 -- 172.21.15.52:0/1980773063 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee54005040 con 0x7fee6c0fbb40 2023-12-16T20:48:37.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.244+0000 7fee6b7fe700 1 --2- 172.21.15.52:0/1980773063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee6c0fbb40 0x7fee6c10e6a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fee5c00b7c0 tx=0x7fee5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:37.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.244+0000 7fee68ff9700 1 -- 172.21.15.52:0/1980773063 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee5c00f6f0 con 0x7fee6c0fbb40 2023-12-16T20:48:37.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.244+0000 7fee71d40700 1 -- 172.21.15.52:0/1980773063 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee6c1176c0 con 0x7fee6c0fbb40 2023-12-16T20:48:37.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.244+0000 7fee68ff9700 1 -- 172.21.15.52:0/1980773063 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee5c00fd00 con 0x7fee6c0fbb40 2023-12-16T20:48:37.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.244+0000 7fee68ff9700 1 -- 172.21.15.52:0/1980773063 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee5c011e90 con 0x7fee6c0fbb40 2023-12-16T20:48:37.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.244+0000 7fee71d40700 1 -- 172.21.15.52:0/1980773063 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee6c117c00 con 0x7fee6c0fbb40 2023-12-16T20:48:37.250 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.245+0000 7fee71d40700 1 -- 172.21.15.52:0/1980773063 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee6c103fb0 con 0x7fee6c0fbb40 2023-12-16T20:48:37.250 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.245+0000 7fee68ff9700 1 -- 172.21.15.52:0/1980773063 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fee5c00f850 con 0x7fee6c0fbb40 2023-12-16T20:48:37.250 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.246+0000 7fee68ff9700 1 --2- 172.21.15.52:0/1980773063 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fee58068f60 0x7fee5806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:37.250 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.246+0000 7fee68ff9700 1 -- 172.21.15.52:0/1980773063 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fee5c086860 con 0x7fee6c0fbb40 2023-12-16T20:48:37.250 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.246+0000 7fee63fff700 1 --2- 172.21.15.52:0/1980773063 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fee58068f60 0x7fee5806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:37.250 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.246+0000 7fee63fff700 1 --2- 172.21.15.52:0/1980773063 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fee58068f60 0x7fee5806b410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fee5400b3b0 tx=0x7fee54009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:37.251 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.250+0000 7fee68ff9700 1 -- 172.21.15.52:0/1980773063 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fee5c053110 con 0x7fee6c0fbb40 2023-12-16T20:48:37.448 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.447+0000 7fee71d40700 1 -- 172.21.15.52:0/1980773063 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee6c0734d0 con 0x7fee6c0fbb40 2023-12-16T20:48:37.448 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.448+0000 7fee68ff9700 1 -- 172.21.15.52:0/1980773063 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+11078 (secure 0 0 0) 0x7fee5c056730 con 0x7fee6c0fbb40 2023-12-16T20:48:37.449 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:48:37.449 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":37,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","created":"2023-12-16T20:44:44.679887+0000","modified":"2023-12-16T20:48:35.123510+0000","last_up_change":"2023-12-16T20:48:33.120930+0000","last_in_change":"2023-12-16T20:48:17.606842+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T20:47:46.398235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"03268674-f4fa-4c69-9fa5-ac2327873b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6803","nonce":362877775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6805","nonce":362877775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6809","nonce":362877775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6807","nonce":362877775}]},"public_addr":"172.21.15.52:6803/362877775","cluster_addr":"172.21.15.52:6805/362877775","heartbeat_back_addr":"172.21.15.52:6809/362877775","heartbeat_front_addr":"172.21.15.52:6807/362877775","state":["exists","up"]},{"osd":1,"uuid":"8cf65e84-7be8-4d6c-872d-d908e7a55c46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6811","nonce":2532880721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6813","nonce":2532880721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6817","nonce":2532880721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6815","nonce":2532880721}]},"public_addr":"172.21.15.52:6811/2532880721","cluster_addr":"172.21.15.52:6813/2532880721","heartbeat_back_addr":"172.21.15.52:6817/2532880721","heartbeat_front_addr":"172.21.15.52:6815/2532880721","state":["exists","up"]},{"osd":2,"uuid":"fd375c62-57d0-4899-aa93-b582a664abbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6819","nonce":4281298930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6821","nonce":4281298930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6825","nonce":4281298930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6823","nonce":4281298930}]},"public_addr":"172.21.15.52:6819/4281298930","cluster_addr":"172.21.15.52:6821/4281298930","heartbeat_back_addr":"172.21.15.52:6825/4281298930","heartbeat_front_addr":"172.21.15.52:6823/4281298930","state":["exists","up"]},{"osd":3,"uuid":"31298367-7548-4ef5-bd2b-a6f586ec14a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6801","nonce":2861125970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6803","nonce":2861125970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6807","nonce":2861125970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6805","nonce":2861125970}]},"public_addr":"172.21.15.150:6801/2861125970","cluster_addr":"172.21.15.150:6803/2861125970","heartbeat_back_addr":"172.21.15.150:6807/2861125970","heartbeat_front_addr":"172.21.15.150:6805/2861125970","state":["exists","up"]},{"osd":4,"uuid":"ab02c027-f21c-4187-af78-f4daaa844db1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6809","nonce":2997946868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6811","nonce":2997946868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6815","nonce":2997946868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6813","nonce":2997946868}]},"public_addr":"172.21.15.150:6809/2997946868","cluster_addr":"172.21.15.150:6811/2997946868","heartbeat_back_addr":"172.21.15.150:6815/2997946868","heartbeat_front_addr":"172.21.15.150:6813/2997946868","state":["exists","up"]},{"osd":5,"uuid":"f62123bf-fcd1-4253-b8c7-78c2379bb90b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6817","nonce":4046379778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6819","nonce":4046379778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6823","nonce":4046379778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6821","nonce":4046379778}]},"public_addr":"172.21.15.150:6817/4046379778","cluster_addr":"172.21.15.150:6819/4046379778","heartbeat_back_addr":"172.21.15.150:6823/4046379778","heartbeat_front_addr":"172.21.15.150:6821/4046379778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:06.709733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:25.479953+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:44.608913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:59.806350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:48:15.471651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:48:31.147029+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6800/2799692744":"2023-12-17T20:46:17.883785+0000","172.21.15.52:0/4160218759":"2023-12-17T20:46:17.883785+0000","172.21.15.52:0/2518812785":"2023-12-17T20:46:17.883785+0000","172.21.15.52:6800/221719009":"2023-12-17T20:45:28.570795+0000","172.21.15.52:0/2730302642":"2023-12-17T20:45:28.570795+0000","172.21.15.52:6801/2799692744":"2023-12-17T20:46:17.883785+0000","172.21.15.52:0/2975128681":"2023-12-17T20:45:28.570795+0000","172.21.15.52:0/806767676":"2023-12-17T20:45:07.199601+0000","172.21.15.52:0/82552375":"2023-12-17T20:45:07.199601+0000","172.21.15.52:0/1810720520":"2023-12-17T20:45:28.570795+0000","172.21.15.52:0/1900058090":"2023-12-17T20:46:17.883785+0000","172.21.15.52:6801/221719009":"2023-12-17T20:45:28.570795+0000","172.21.15.52:6800/2740978054":"2023-12-17T20:45:07.199601+0000","172.21.15.52:6801/2740978054":"2023-12-17T20:45:07.199601+0000","172.21.15.52:0/27878571":"2023-12-17T20:45:07.199601+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T20:48:37.451 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.450+0000 7fee71d40700 1 -- 172.21.15.52:0/1980773063 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fee58068f60 msgr2=0x7fee5806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:37.451 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.450+0000 7fee71d40700 1 --2- 172.21.15.52:0/1980773063 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fee58068f60 0x7fee5806b410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fee5400b3b0 tx=0x7fee54009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:37.451 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.450+0000 7fee71d40700 1 -- 172.21.15.52:0/1980773063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee6c0fbb40 msgr2=0x7fee6c10e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:37.451 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.450+0000 7fee71d40700 1 --2- 172.21.15.52:0/1980773063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee6c0fbb40 0x7fee6c10e6a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fee5c00b7c0 tx=0x7fee5c009250 comp rx=0 tx=0).stop 2023-12-16T20:48:37.451 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.450+0000 7fee71d40700 1 -- 172.21.15.52:0/1980773063 shutdown_connections 2023-12-16T20:48:37.452 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.450+0000 7fee71d40700 1 --2- 172.21.15.52:0/1980773063 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fee58068f60 0x7fee5806b410 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:37.452 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.450+0000 7fee71d40700 1 --2- 172.21.15.52:0/1980773063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee6c0fbb40 0x7fee6c10e6a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:37.452 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.450+0000 7fee71d40700 1 --2- 172.21.15.52:0/1980773063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6c101b00 0x7fee6c10ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:37.452 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.450+0000 7fee71d40700 1 -- 172.21.15.52:0/1980773063 >> 172.21.15.52:0/1980773063 conn(0x7fee6c0719b0 msgr2=0x7fee6c0faa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:37.452 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.450+0000 7fee71d40700 1 -- 172.21.15.52:0/1980773063 shutdown_connections 2023-12-16T20:48:37.452 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:37.451+0000 7fee71d40700 1 -- 172.21.15.52:0/1980773063 wait complete. 2023-12-16T20:48:37.902 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T20:47:46.398235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-16T20:48:37.902 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph osd pool get .mgr pg_num 2023-12-16T20:48:38.168 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:38.544 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:38 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/1980773063' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T20:48:38.544 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:38 smithi052 ceph-mon[80761]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:38.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:38 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/1980773063' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T20:48:38.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:38 smithi150 ceph-mon[88000]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:39.044 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.042+0000 7f4fd64c0700 1 -- 172.21.15.52:0/1248957390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fd00fbb00 msgr2=0x7f4fd00fbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:39.044 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.042+0000 7f4fd64c0700 1 --2- 172.21.15.52:0/1248957390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fd00fbb00 0x7f4fd00fbf70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4004940 tx=0x7f4fc4009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:39.044 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.042+0000 7f4fd64c0700 1 -- 172.21.15.52:0/1248957390 shutdown_connections 2023-12-16T20:48:39.044 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.042+0000 7f4fd64c0700 1 --2- 172.21.15.52:0/1248957390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fd00fbb00 0x7f4fd00fbf70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:39.044 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.042+0000 7f4fd64c0700 1 --2- 172.21.15.52:0/1248957390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fd0101b60 0x7f4fd0101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:39.044 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.042+0000 7f4fd64c0700 1 -- 172.21.15.52:0/1248957390 >> 172.21.15.52:0/1248957390 conn(0x7f4fd00f7680 msgr2=0x7f4fd00f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:39.044 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.042+0000 7f4fd64c0700 1 -- 172.21.15.52:0/1248957390 shutdown_connections 2023-12-16T20:48:39.044 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.042+0000 7f4fd64c0700 1 -- 172.21.15.52:0/1248957390 wait complete. 2023-12-16T20:48:39.045 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.043+0000 7f4fd64c0700 1 Processor -- start 2023-12-16T20:48:39.045 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.043+0000 7f4fd64c0700 1 -- start start 2023-12-16T20:48:39.045 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.043+0000 7f4fd64c0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fd00fbb00 0x7f4fd010e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:39.045 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.043+0000 7f4fd64c0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fd0101b60 0x7f4fd010ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:39.045 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.043+0000 7f4fd64c0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd010f2b0 con 0x7f4fd00fbb00 2023-12-16T20:48:39.045 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.043+0000 7f4fd64c0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd010f3f0 con 0x7f4fd0101b60 2023-12-16T20:48:39.045 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.043+0000 7f4fcffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fd00fbb00 0x7f4fd010e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:39.045 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.043+0000 7f4fcffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fd00fbb00 0x7f4fd010e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42616/0 (socket says 172.21.15.52:42616) 2023-12-16T20:48:39.045 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.043+0000 7f4fcffff700 1 -- 172.21.15.52:0/1695611259 learned_addr learned my addr 172.21.15.52:0/1695611259 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:39.046 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.044+0000 7f4fcf7fe700 1 --2- 172.21.15.52:0/1695611259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fd0101b60 0x7f4fd010ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:39.046 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.044+0000 7f4fcffff700 1 -- 172.21.15.52:0/1695611259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fd0101b60 msgr2=0x7f4fd010ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:39.046 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.044+0000 7f4fcffff700 1 --2- 172.21.15.52:0/1695611259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fd0101b60 0x7f4fd010ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:39.046 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.044+0000 7f4fcffff700 1 -- 172.21.15.52:0/1695611259 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc4005040 con 0x7f4fd00fbb00 2023-12-16T20:48:39.046 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.044+0000 7f4fcffff700 1 --2- 172.21.15.52:0/1695611259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fd00fbb00 0x7f4fd010e750 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000b7c0 tx=0x7f4fc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:39.046 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.044+0000 7f4fccff9700 1 -- 172.21.15.52:0/1695611259 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fc000f6f0 con 0x7f4fd00fbb00 2023-12-16T20:48:39.046 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.044+0000 7f4fccff9700 1 -- 172.21.15.52:0/1695611259 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4fc000fd00 con 0x7f4fd00fbb00 2023-12-16T20:48:39.047 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.044+0000 7f4fd64c0700 1 -- 172.21.15.52:0/1695611259 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fd0117720 con 0x7f4fd00fbb00 2023-12-16T20:48:39.047 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.044+0000 7f4fccff9700 1 -- 172.21.15.52:0/1695611259 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fc0011e90 con 0x7f4fd00fbb00 2023-12-16T20:48:39.047 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.044+0000 7f4fd64c0700 1 -- 172.21.15.52:0/1695611259 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fd0117c60 con 0x7f4fd00fbb00 2023-12-16T20:48:39.047 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.045+0000 7f4fd64c0700 1 -- 172.21.15.52:0/1695611259 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fd0103f40 con 0x7f4fd00fbb00 2023-12-16T20:48:39.047 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.046+0000 7f4fccff9700 1 -- 172.21.15.52:0/1695611259 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f4fc000f850 con 0x7f4fd00fbb00 2023-12-16T20:48:39.047 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.046+0000 7f4fccff9700 1 --2- 172.21.15.52:0/1695611259 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4fb8068f60 0x7f4fb806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:39.048 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.046+0000 7f4fccff9700 1 -- 172.21.15.52:0/1695611259 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4fc0086900 con 0x7f4fd00fbb00 2023-12-16T20:48:39.048 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.046+0000 7f4fcf7fe700 1 --2- 172.21.15.52:0/1695611259 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4fb8068f60 0x7f4fb806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:39.048 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.047+0000 7f4fcf7fe700 1 --2- 172.21.15.52:0/1695611259 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4fb8068f60 0x7f4fb806b410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4004f90 tx=0x7f4fc4009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:39.051 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.051+0000 7f4fccff9700 1 -- 172.21.15.52:0/1695611259 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4fc0053110 con 0x7f4fd00fbb00 2023-12-16T20:48:39.246 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.245+0000 7f4fd64c0700 1 -- 172.21.15.52:0/1695611259 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4fd0066400 con 0x7f4fd00fbb00 2023-12-16T20:48:39.246 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.245+0000 7f4fccff9700 1 -- 172.21.15.52:0/1695611259 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4fc0056730 con 0x7f4fd00fbb00 2023-12-16T20:48:39.246 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 1 2023-12-16T20:48:39.248 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.247+0000 7f4fd64c0700 1 -- 172.21.15.52:0/1695611259 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4fb8068f60 msgr2=0x7f4fb806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:39.248 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.247+0000 7f4fd64c0700 1 --2- 172.21.15.52:0/1695611259 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4fb8068f60 0x7f4fb806b410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4004f90 tx=0x7f4fc4009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:39.248 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.247+0000 7f4fd64c0700 1 -- 172.21.15.52:0/1695611259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fd00fbb00 msgr2=0x7f4fd010e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:39.248 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.247+0000 7f4fd64c0700 1 --2- 172.21.15.52:0/1695611259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fd00fbb00 0x7f4fd010e750 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000b7c0 tx=0x7f4fc0009250 comp rx=0 tx=0).stop 2023-12-16T20:48:39.248 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.247+0000 7f4fd64c0700 1 -- 172.21.15.52:0/1695611259 shutdown_connections 2023-12-16T20:48:39.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.247+0000 7f4fd64c0700 1 --2- 172.21.15.52:0/1695611259 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4fb8068f60 0x7f4fb806b410 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:39.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.247+0000 7f4fd64c0700 1 --2- 172.21.15.52:0/1695611259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fd00fbb00 0x7f4fd010e750 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:39.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.247+0000 7f4fd64c0700 1 --2- 172.21.15.52:0/1695611259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fd0101b60 0x7f4fd010ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:39.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.247+0000 7f4fd64c0700 1 -- 172.21.15.52:0/1695611259 >> 172.21.15.52:0/1695611259 conn(0x7f4fd00f7680 msgr2=0x7f4fd00f8200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:39.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.247+0000 7f4fd64c0700 1 -- 172.21.15.52:0/1695611259 shutdown_connections 2023-12-16T20:48:39.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:39.247+0000 7f4fd64c0700 1 -- 172.21.15.52:0/1695611259 wait complete. 2023-12-16T20:48:39.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:39 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/1695611259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-16T20:48:39.535 INFO:tasks.cephadm:Setting up client nodes... 2023-12-16T20:48:39.535 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-12-16T20:48:39.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:39 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/1695611259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-16T20:48:39.800 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:40.576 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:40 smithi052 ceph-mon[80761]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:40.686 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.685+0000 7f10a3773700 1 -- 172.21.15.52:0/622214578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f109c0fcbf0 msgr2=0x7f109c0fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:40.686 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.685+0000 7f10a3773700 1 --2- 172.21.15.52:0/622214578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f109c0fcbf0 0x7f109c0fd000 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f108c009ce0 tx=0x7f108c004d10 comp rx=0 tx=0).stop 2023-12-16T20:48:40.686 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.685+0000 7f10a3773700 1 -- 172.21.15.52:0/622214578 shutdown_connections 2023-12-16T20:48:40.686 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.685+0000 7f10a3773700 1 --2- 172.21.15.52:0/622214578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f109c0fde00 0x7f109c0fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:40.686 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.685+0000 7f10a3773700 1 --2- 172.21.15.52:0/622214578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f109c0fcbf0 0x7f109c0fd000 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:40.686 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.685+0000 7f10a3773700 1 -- 172.21.15.52:0/622214578 >> 172.21.15.52:0/622214578 conn(0x7f109c0f82d0 msgr2=0x7f109c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:40.686 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.685+0000 7f10a3773700 1 -- 172.21.15.52:0/622214578 shutdown_connections 2023-12-16T20:48:40.686 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.685+0000 7f10a3773700 1 -- 172.21.15.52:0/622214578 wait complete. 2023-12-16T20:48:40.688 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.686+0000 7f10a3773700 1 Processor -- start 2023-12-16T20:48:40.688 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.686+0000 7f10a3773700 1 -- start start 2023-12-16T20:48:40.688 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.686+0000 7f10a3773700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f109c0fcbf0 0x7f109c10d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:40.688 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.686+0000 7f10a3773700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f109c0fde00 0x7f109c10dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:40.688 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.686+0000 7f10a3773700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109c10e290 con 0x7f109c0fcbf0 2023-12-16T20:48:40.688 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.686+0000 7f10a3773700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109c10ead0 con 0x7f109c0fde00 2023-12-16T20:48:40.688 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.686+0000 7f10a150f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f109c0fcbf0 0x7f109c10d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:40.688 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.686+0000 7f10a150f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f109c0fcbf0 0x7f109c10d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42634/0 (socket says 172.21.15.52:42634) 2023-12-16T20:48:40.688 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.686+0000 7f10a150f700 1 -- 172.21.15.52:0/3721812950 learned_addr learned my addr 172.21.15.52:0/3721812950 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:40.688 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.686+0000 7f10a0d0e700 1 --2- 172.21.15.52:0/3721812950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f109c0fde00 0x7f109c10dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:40.689 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.686+0000 7f10a150f700 1 -- 172.21.15.52:0/3721812950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f109c0fde00 msgr2=0x7f109c10dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:40.689 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.686+0000 7f10a150f700 1 --2- 172.21.15.52:0/3721812950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f109c0fde00 0x7f109c10dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:40.689 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.686+0000 7f10a150f700 1 -- 172.21.15.52:0/3721812950 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108c009940 con 0x7f109c0fcbf0 2023-12-16T20:48:40.689 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.686+0000 7f10a150f700 1 --2- 172.21.15.52:0/3721812950 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f109c0fcbf0 0x7f109c10d6c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f108c009900 tx=0x7f108c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:40.689 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.687+0000 7f1099ffb700 1 -- 172.21.15.52:0/3721812950 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f108c00e040 con 0x7f109c0fcbf0 2023-12-16T20:48:40.689 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.687+0000 7f10a3773700 1 -- 172.21.15.52:0/3721812950 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f109c10ed50 con 0x7f109c0fcbf0 2023-12-16T20:48:40.689 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.687+0000 7f1099ffb700 1 -- 172.21.15.52:0/3721812950 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f108c0128c0 con 0x7f109c0fcbf0 2023-12-16T20:48:40.689 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.687+0000 7f1099ffb700 1 -- 172.21.15.52:0/3721812950 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f108c01c7a0 con 0x7f109c0fcbf0 2023-12-16T20:48:40.689 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.687+0000 7f10a3773700 1 -- 172.21.15.52:0/3721812950 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f109c10f290 con 0x7f109c0fcbf0 2023-12-16T20:48:40.690 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.688+0000 7f10a3773700 1 -- 172.21.15.52:0/3721812950 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f109c04bb80 con 0x7f109c0fcbf0 2023-12-16T20:48:40.690 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.688+0000 7f1099ffb700 1 -- 172.21.15.52:0/3721812950 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f108c012a30 con 0x7f109c0fcbf0 2023-12-16T20:48:40.690 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.689+0000 7f1099ffb700 1 --2- 172.21.15.52:0/3721812950 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1084068f60 0x7f108406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:40.690 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.689+0000 7f1099ffb700 1 -- 172.21.15.52:0/3721812950 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f108c088840 con 0x7f109c0fcbf0 2023-12-16T20:48:40.690 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.689+0000 7f10a0d0e700 1 --2- 172.21.15.52:0/3721812950 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1084068f60 0x7f108406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:40.690 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.689+0000 7f10a0d0e700 1 --2- 172.21.15.52:0/3721812950 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1084068f60 0x7f108406b410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1090009bf0 tx=0x7f10900091a0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:40.694 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.693+0000 7f1099ffb700 1 -- 172.21.15.52:0/3721812950 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f108c058a40 con 0x7f109c0fcbf0 2023-12-16T20:48:40.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:40 smithi150 ceph-mon[88000]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:40.969 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.968+0000 7f10a3773700 1 -- 172.21.15.52:0/3721812950 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f109c105610 con 0x7f109c0fcbf0 2023-12-16T20:48:40.973 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.972+0000 7f1099ffb700 1 -- 172.21.15.52:0/3721812950 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f108c0585d0 con 0x7f109c0fcbf0 2023-12-16T20:48:40.973 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2023-12-16T20:48:40.973 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQAoDX5loKrUORAApIh74ssEGHlQGQNUhWASCg== 2023-12-16T20:48:40.975 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.974+0000 7f10a3773700 1 -- 172.21.15.52:0/3721812950 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1084068f60 msgr2=0x7f108406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:40.975 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.974+0000 7f10a3773700 1 --2- 172.21.15.52:0/3721812950 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1084068f60 0x7f108406b410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1090009bf0 tx=0x7f10900091a0 comp rx=0 tx=0).stop 2023-12-16T20:48:40.975 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.974+0000 7f10a3773700 1 -- 172.21.15.52:0/3721812950 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f109c0fcbf0 msgr2=0x7f109c10d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:40.975 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.974+0000 7f10a3773700 1 --2- 172.21.15.52:0/3721812950 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f109c0fcbf0 0x7f109c10d6c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f108c009900 tx=0x7f108c004d10 comp rx=0 tx=0).stop 2023-12-16T20:48:40.976 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.974+0000 7f10a3773700 1 -- 172.21.15.52:0/3721812950 shutdown_connections 2023-12-16T20:48:40.976 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.974+0000 7f10a3773700 1 --2- 172.21.15.52:0/3721812950 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1084068f60 0x7f108406b410 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:40.976 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.974+0000 7f10a3773700 1 --2- 172.21.15.52:0/3721812950 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f109c0fcbf0 0x7f109c10d6c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:40.976 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.974+0000 7f10a3773700 1 --2- 172.21.15.52:0/3721812950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f109c0fde00 0x7f109c10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:40.976 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.974+0000 7f10a3773700 1 -- 172.21.15.52:0/3721812950 >> 172.21.15.52:0/3721812950 conn(0x7f109c0f82d0 msgr2=0x7f109c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:40.976 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.975+0000 7f10a3773700 1 -- 172.21.15.52:0/3721812950 shutdown_connections 2023-12-16T20:48:40.976 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:40.975+0000 7f10a3773700 1 -- 172.21.15.52:0/3721812950 wait complete. 2023-12-16T20:48:41.377 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:48:41.377 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-12-16T20:48:41.377 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-16T20:48:41.426 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-12-16T20:48:41.692 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi150/config 2023-12-16T20:48:41.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:41 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3721812950' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T20:48:41.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:41 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3721812950' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-16T20:48:41.754 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:41 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/3721812950' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T20:48:41.754 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:41 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/3721812950' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-16T20:48:42.406 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.406+0000 7fd5af51f700 1 -- 172.21.15.150:0/2527538078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a80faed0 msgr2=0x7fd5a80fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:42.406 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.406+0000 7fd5af51f700 1 --2- 172.21.15.150:0/2527538078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a80faed0 0x7fd5a80fb2e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4004940 tx=0x7fd5a4009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:42.406 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.406+0000 7fd5af51f700 1 -- 172.21.15.150:0/2527538078 shutdown_connections 2023-12-16T20:48:42.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.406+0000 7fd5af51f700 1 --2- 172.21.15.150:0/2527538078 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5a80fdf60 0x7fd5a80fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:42.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.406+0000 7fd5af51f700 1 --2- 172.21.15.150:0/2527538078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a80faed0 0x7fd5a80fb2e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:42.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.406+0000 7fd5af51f700 1 -- 172.21.15.150:0/2527538078 >> 172.21.15.150:0/2527538078 conn(0x7fd5a80f6950 msgr2=0x7fd5a80f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:42.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.406+0000 7fd5af51f700 1 -- 172.21.15.150:0/2527538078 shutdown_connections 2023-12-16T20:48:42.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.406+0000 7fd5af51f700 1 -- 172.21.15.150:0/2527538078 wait complete. 2023-12-16T20:48:42.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.406+0000 7fd5af51f700 1 Processor -- start 2023-12-16T20:48:42.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.406+0000 7fd5af51f700 1 -- start start 2023-12-16T20:48:42.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.406+0000 7fd5af51f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5a80faed0 0x7fd5a810d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:42.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.406+0000 7fd5af51f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a80fdf60 0x7fd5a810da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:42.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.406+0000 7fd5af51f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a810e080 con 0x7fd5a80faed0 2023-12-16T20:48:42.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.406+0000 7fd5af51f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a810e920 con 0x7fd5a80fdf60 2023-12-16T20:48:42.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.407+0000 7fd5acaba700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a80fdf60 0x7fd5a810da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:42.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.407+0000 7fd5acaba700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a80fdf60 0x7fd5a810da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41196/0 (socket says 172.21.15.150:41196) 2023-12-16T20:48:42.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.407+0000 7fd5acaba700 1 -- 172.21.15.150:0/3217290799 learned_addr learned my addr 172.21.15.150:0/3217290799 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-12-16T20:48:42.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.407+0000 7fd5ad2bb700 1 --2- 172.21.15.150:0/3217290799 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5a80faed0 0x7fd5a810d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:42.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.407+0000 7fd5acaba700 1 -- 172.21.15.150:0/3217290799 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5a80faed0 msgr2=0x7fd5a810d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:42.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.407+0000 7fd5acaba700 1 --2- 172.21.15.150:0/3217290799 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5a80faed0 0x7fd5a810d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:42.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.407+0000 7fd5acaba700 1 -- 172.21.15.150:0/3217290799 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a4005040 con 0x7fd5a80fdf60 2023-12-16T20:48:42.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.407+0000 7fd5acaba700 1 --2- 172.21.15.150:0/3217290799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a80fdf60 0x7fd5a810da80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd59800eb40 tx=0x7fd59800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:42.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.407+0000 7fd59dffb700 1 -- 172.21.15.150:0/3217290799 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd598007cc0 con 0x7fd5a80fdf60 2023-12-16T20:48:42.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.407+0000 7fd5af51f700 1 -- 172.21.15.150:0/3217290799 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a810eba0 con 0x7fd5a80fdf60 2023-12-16T20:48:42.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.407+0000 7fd59dffb700 1 -- 172.21.15.150:0/3217290799 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd59800f040 con 0x7fd5a80fdf60 2023-12-16T20:48:42.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.407+0000 7fd59dffb700 1 -- 172.21.15.150:0/3217290799 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd59800c990 con 0x7fd5a80fdf60 2023-12-16T20:48:42.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.408+0000 7fd5af51f700 1 -- 172.21.15.150:0/3217290799 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a810f0e0 con 0x7fd5a80fdf60 2023-12-16T20:48:42.410 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.409+0000 7fd5af51f700 1 -- 172.21.15.150:0/3217290799 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a804bb80 con 0x7fd5a80fdf60 2023-12-16T20:48:42.410 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.409+0000 7fd59dffb700 1 -- 172.21.15.150:0/3217290799 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fd59800cc00 con 0x7fd5a80fdf60 2023-12-16T20:48:42.410 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.409+0000 7fd59dffb700 1 --2- 172.21.15.150:0/3217290799 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd594068f60 0x7fd59406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:42.410 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.410+0000 7fd59dffb700 1 -- 172.21.15.150:0/3217290799 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd598086650 con 0x7fd5a80fdf60 2023-12-16T20:48:42.410 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.410+0000 7fd5ad2bb700 1 --2- 172.21.15.150:0/3217290799 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd594068f60 0x7fd59406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:42.411 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.410+0000 7fd5ad2bb700 1 --2- 172.21.15.150:0/3217290799 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd594068f60 0x7fd59406b410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4000c00 tx=0x7fd5a4009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:42.414 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.414+0000 7fd59dffb700 1 -- 172.21.15.150:0/3217290799 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd59804da00 con 0x7fd5a80fdf60 2023-12-16T20:48:42.674 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.674+0000 7fd5af51f700 1 -- 172.21.15.150:0/3217290799 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fd5a81055b0 con 0x7fd5a80fdf60 2023-12-16T20:48:42.674 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:42 smithi150 ceph-mon[88000]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:42.678 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.678+0000 7fd59dffb700 1 -- 172.21.15.150:0/3217290799 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fd598056920 con 0x7fd5a80fdf60 2023-12-16T20:48:42.679 INFO:teuthology.orchestra.run.smithi150.stdout:[client.1] 2023-12-16T20:48:42.679 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAqDX5lIBVEKBAAjF1SODdmhp6axaMjB0w52g== 2023-12-16T20:48:42.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.680+0000 7fd5af51f700 1 -- 172.21.15.150:0/3217290799 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd594068f60 msgr2=0x7fd59406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:42.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.680+0000 7fd5af51f700 1 --2- 172.21.15.150:0/3217290799 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd594068f60 0x7fd59406b410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4000c00 tx=0x7fd5a4009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:42.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.680+0000 7fd5af51f700 1 -- 172.21.15.150:0/3217290799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a80fdf60 msgr2=0x7fd5a810da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:42.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.680+0000 7fd5af51f700 1 --2- 172.21.15.150:0/3217290799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a80fdf60 0x7fd5a810da80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd59800eb40 tx=0x7fd59800b040 comp rx=0 tx=0).stop 2023-12-16T20:48:42.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.680+0000 7fd5af51f700 1 -- 172.21.15.150:0/3217290799 shutdown_connections 2023-12-16T20:48:42.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.680+0000 7fd5af51f700 1 --2- 172.21.15.150:0/3217290799 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd594068f60 0x7fd59406b410 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:42.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.680+0000 7fd5af51f700 1 --2- 172.21.15.150:0/3217290799 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5a80faed0 0x7fd5a810d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:42.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.680+0000 7fd5af51f700 1 --2- 172.21.15.150:0/3217290799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a80fdf60 0x7fd5a810da80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:42.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.681+0000 7fd5af51f700 1 -- 172.21.15.150:0/3217290799 >> 172.21.15.150:0/3217290799 conn(0x7fd5a80f6950 msgr2=0x7fd5a8101190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:42.682 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.681+0000 7fd5af51f700 1 -- 172.21.15.150:0/3217290799 shutdown_connections 2023-12-16T20:48:42.682 INFO:teuthology.orchestra.run.smithi150.stderr:2023-12-16T20:48:42.681+0000 7fd5af51f700 1 -- 172.21.15.150:0/3217290799 wait complete. 2023-12-16T20:48:42.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:42 smithi052 ceph-mon[80761]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:42.911 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-16T20:48:42.911 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2023-12-16T20:48:42.911 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-16T20:48:42.953 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-16T20:48:42.953 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-12-16T20:48:42.954 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph mgr dump --format=json 2023-12-16T20:48:43.230 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:43.568 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:43 smithi052 ceph-mon[80761]: from='client.? 172.21.15.150:0/3217290799' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T20:48:43.568 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:43 smithi052 ceph-mon[80761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T20:48:43.568 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:43 smithi052 ceph-mon[80761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-16T20:48:43.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:43 smithi150 ceph-mon[88000]: from='client.? 172.21.15.150:0/3217290799' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T20:48:43.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:43 smithi150 ceph-mon[88000]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T20:48:43.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:43 smithi150 ceph-mon[88000]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-16T20:48:44.084 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.081+0000 7fb9f4c61700 1 -- 172.21.15.52:0/2262272864 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9f00fe5a0 msgr2=0x7fb9f0106c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:44.084 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.081+0000 7fb9f4c61700 1 --2- 172.21.15.52:0/2262272864 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9f00fe5a0 0x7fb9f0106c70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8004940 tx=0x7fb9d8009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:44.084 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.082+0000 7fb9f4c61700 1 -- 172.21.15.52:0/2262272864 shutdown_connections 2023-12-16T20:48:44.084 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.082+0000 7fb9f4c61700 1 --2- 172.21.15.52:0/2262272864 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9f00fe5a0 0x7fb9f0106c70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:44.084 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.082+0000 7fb9f4c61700 1 --2- 172.21.15.52:0/2262272864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9f00fdc00 0x7fb9f00fdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:44.084 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.082+0000 7fb9f4c61700 1 -- 172.21.15.52:0/2262272864 >> 172.21.15.52:0/2262272864 conn(0x7fb9f00f76c0 msgr2=0x7fb9f00f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:44.084 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.082+0000 7fb9f4c61700 1 -- 172.21.15.52:0/2262272864 shutdown_connections 2023-12-16T20:48:44.084 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.082+0000 7fb9f4c61700 1 -- 172.21.15.52:0/2262272864 wait complete. 2023-12-16T20:48:44.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.082+0000 7fb9f4c61700 1 Processor -- start 2023-12-16T20:48:44.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.082+0000 7fb9f4c61700 1 -- start start 2023-12-16T20:48:44.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.082+0000 7fb9f4c61700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9f00fdc00 0x7fb9f0113660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:44.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.082+0000 7fb9f4c61700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9f00fe5a0 0x7fb9f0113be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:44.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.082+0000 7fb9f4c61700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f010d520 con 0x7fb9f00fe5a0 2023-12-16T20:48:44.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.082+0000 7fb9f4c61700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f010d690 con 0x7fb9f00fdc00 2023-12-16T20:48:44.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.083+0000 7fb9edd9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9f00fe5a0 0x7fb9f0113be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:44.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.083+0000 7fb9edd9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9f00fe5a0 0x7fb9f0113be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38146/0 (socket says 172.21.15.52:38146) 2023-12-16T20:48:44.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.083+0000 7fb9edd9b700 1 -- 172.21.15.52:0/4212763248 learned_addr learned my addr 172.21.15.52:0/4212763248 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:44.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.083+0000 7fb9edd9b700 1 -- 172.21.15.52:0/4212763248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9f00fdc00 msgr2=0x7fb9f0113660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:44.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.083+0000 7fb9edd9b700 1 --2- 172.21.15.52:0/4212763248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9f00fdc00 0x7fb9f0113660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:44.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.083+0000 7fb9edd9b700 1 -- 172.21.15.52:0/4212763248 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d8005040 con 0x7fb9f00fe5a0 2023-12-16T20:48:44.085 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.083+0000 7fb9edd9b700 1 --2- 172.21.15.52:0/4212763248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9f00fe5a0 0x7fb9f0113be0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800b3b0 tx=0x7fb9d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:44.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.083+0000 7fb9e6ffd700 1 -- 172.21.15.52:0/4212763248 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9d800e040 con 0x7fb9f00fe5a0 2023-12-16T20:48:44.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.083+0000 7fb9e6ffd700 1 -- 172.21.15.52:0/4212763248 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9d8004290 con 0x7fb9f00fe5a0 2023-12-16T20:48:44.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.083+0000 7fb9f4c61700 1 -- 172.21.15.52:0/4212763248 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f010d910 con 0x7fb9f00fe5a0 2023-12-16T20:48:44.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.083+0000 7fb9e6ffd700 1 -- 172.21.15.52:0/4212763248 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9d801c3f0 con 0x7fb9f00fe5a0 2023-12-16T20:48:44.086 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.083+0000 7fb9f4c61700 1 -- 172.21.15.52:0/4212763248 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f010de50 con 0x7fb9f00fe5a0 2023-12-16T20:48:44.087 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.084+0000 7fb9f4c61700 1 -- 172.21.15.52:0/4212763248 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f01042d0 con 0x7fb9f00fe5a0 2023-12-16T20:48:44.087 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.085+0000 7fb9e6ffd700 1 -- 172.21.15.52:0/4212763248 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb9d800bc50 con 0x7fb9f00fe5a0 2023-12-16T20:48:44.087 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.085+0000 7fb9e6ffd700 1 --2- 172.21.15.52:0/4212763248 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb9dc068f60 0x7fb9dc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:44.087 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.085+0000 7fb9e6ffd700 1 -- 172.21.15.52:0/4212763248 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb9d8087de0 con 0x7fb9f00fe5a0 2023-12-16T20:48:44.087 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.085+0000 7fb9ee59c700 1 --2- 172.21.15.52:0/4212763248 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb9dc068f60 0x7fb9dc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:44.087 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.086+0000 7fb9ee59c700 1 --2- 172.21.15.52:0/4212763248 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb9dc068f60 0x7fb9dc06b410 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00fa120 tx=0x7fb9e8009370 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:44.090 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.090+0000 7fb9e6ffd700 1 -- 172.21.15.52:0/4212763248 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb9d8058120 con 0x7fb9f00fe5a0 2023-12-16T20:48:44.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.344+0000 7fb9f4c61700 1 -- 172.21.15.52:0/4212763248 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fb9f004bb80 con 0x7fb9f00fe5a0 2023-12-16T20:48:44.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.347+0000 7fb9e6ffd700 1 -- 172.21.15.52:0/4212763248 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+166776 (secure 0 0 0) 0x7fb9d8057cb0 con 0x7fb9f00fe5a0 2023-12-16T20:48:44.352 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:48:44.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.350+0000 7fb9f4c61700 1 -- 172.21.15.52:0/4212763248 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb9dc068f60 msgr2=0x7fb9dc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:44.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.350+0000 7fb9f4c61700 1 --2- 172.21.15.52:0/4212763248 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb9dc068f60 0x7fb9dc06b410 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00fa120 tx=0x7fb9e8009370 comp rx=0 tx=0).stop 2023-12-16T20:48:44.354 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.350+0000 7fb9f4c61700 1 -- 172.21.15.52:0/4212763248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9f00fe5a0 msgr2=0x7fb9f0113be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:44.354 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.350+0000 7fb9f4c61700 1 --2- 172.21.15.52:0/4212763248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9f00fe5a0 0x7fb9f0113be0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800b3b0 tx=0x7fb9d8009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:44.354 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.350+0000 7fb9f4c61700 1 -- 172.21.15.52:0/4212763248 shutdown_connections 2023-12-16T20:48:44.355 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.350+0000 7fb9f4c61700 1 --2- 172.21.15.52:0/4212763248 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb9dc068f60 0x7fb9dc06b410 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:44.355 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.350+0000 7fb9f4c61700 1 --2- 172.21.15.52:0/4212763248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9f00fdc00 0x7fb9f0113660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:44.355 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.350+0000 7fb9f4c61700 1 --2- 172.21.15.52:0/4212763248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9f00fe5a0 0x7fb9f0113be0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:44.355 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.350+0000 7fb9f4c61700 1 -- 172.21.15.52:0/4212763248 >> 172.21.15.52:0/4212763248 conn(0x7fb9f00f76c0 msgr2=0x7fb9f0101450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:44.355 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.350+0000 7fb9f4c61700 1 -- 172.21.15.52:0/4212763248 shutdown_connections 2023-12-16T20:48:44.355 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:44.350+0000 7fb9f4c61700 1 -- 172.21.15.52:0/4212763248 wait complete. 2023-12-16T20:48:44.608 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:44 smithi052 ceph-mon[80761]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:44.609 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:44 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/4212763248' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-12-16T20:48:44.634 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":19,"active_gid":14214,"active_name":"smithi052.nmkcbs","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":2056060258},{"type":"v1","addr":"172.21.15.52:6801","nonce":2056060258}]},"active_addr":"172.21.15.52:6801/2056060258","active_change":"2023-12-16T20:46:17.883912+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14236,"name":"smithi150.vjiupe","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.52:8443/","prometheus":"http://172.21.15.52:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":4102077161}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":1564231530}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":484990560}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":3105593018}]}]} 2023-12-16T20:48:44.638 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-12-16T20:48:44.638 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-12-16T20:48:44.638 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-16T20:48:44.903 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:44.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:44 smithi150 ceph-mon[88000]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:44.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:44 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/4212763248' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-12-16T20:48:45.785 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.783+0000 7f68ba97a700 1 -- 172.21.15.52:0/2675584200 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68b40fc320 msgr2=0x7f68b40fc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:45.786 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.783+0000 7f68ba97a700 1 --2- 172.21.15.52:0/2675584200 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68b40fc320 0x7f68b40fc790 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f68a8004940 tx=0x7f68a8009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:45.786 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.783+0000 7f68ba97a700 1 -- 172.21.15.52:0/2675584200 shutdown_connections 2023-12-16T20:48:45.786 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.783+0000 7f68ba97a700 1 --2- 172.21.15.52:0/2675584200 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68b40fc320 0x7f68b40fc790 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:45.786 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.783+0000 7f68ba97a700 1 --2- 172.21.15.52:0/2675584200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68b4101c40 0x7f68b4102010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:45.786 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.783+0000 7f68ba97a700 1 -- 172.21.15.52:0/2675584200 >> 172.21.15.52:0/2675584200 conn(0x7f68b40711b0 msgr2=0x7f68b40715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:45.786 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.783+0000 7f68ba97a700 1 -- 172.21.15.52:0/2675584200 shutdown_connections 2023-12-16T20:48:45.786 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.783+0000 7f68ba97a700 1 -- 172.21.15.52:0/2675584200 wait complete. 2023-12-16T20:48:45.786 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.784+0000 7f68ba97a700 1 Processor -- start 2023-12-16T20:48:45.786 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.784+0000 7f68ba97a700 1 -- start start 2023-12-16T20:48:45.786 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.784+0000 7f68ba97a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68b40fc320 0x7f68b410e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:45.786 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.784+0000 7f68ba97a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68b4101c40 0x7f68b410ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:45.786 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.784+0000 7f68ba97a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b410f2e0 con 0x7f68b4101c40 2023-12-16T20:48:45.786 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.784+0000 7f68ba97a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b410f420 con 0x7f68b40fc320 2023-12-16T20:48:45.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.784+0000 7f68b37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68b4101c40 0x7f68b410ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:45.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.784+0000 7f68b37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68b4101c40 0x7f68b410ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38172/0 (socket says 172.21.15.52:38172) 2023-12-16T20:48:45.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.784+0000 7f68b37fe700 1 -- 172.21.15.52:0/4176613030 learned_addr learned my addr 172.21.15.52:0/4176613030 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:45.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.784+0000 7f68b3fff700 1 --2- 172.21.15.52:0/4176613030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68b40fc320 0x7f68b410e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:45.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.784+0000 7f68b37fe700 1 -- 172.21.15.52:0/4176613030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68b40fc320 msgr2=0x7f68b410e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:45.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.784+0000 7f68b37fe700 1 --2- 172.21.15.52:0/4176613030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68b40fc320 0x7f68b410e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:45.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.784+0000 7f68b37fe700 1 -- 172.21.15.52:0/4176613030 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a8005040 con 0x7f68b4101c40 2023-12-16T20:48:45.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.785+0000 7f68b37fe700 1 --2- 172.21.15.52:0/4176613030 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68b4101c40 0x7f68b410ecc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009100 tx=0x7f68a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:45.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.785+0000 7f68b0ff9700 1 -- 172.21.15.52:0/4176613030 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68a800e040 con 0x7f68b4101c40 2023-12-16T20:48:45.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.785+0000 7f68ba97a700 1 -- 172.21.15.52:0/4176613030 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b41176f0 con 0x7f68b4101c40 2023-12-16T20:48:45.788 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.785+0000 7f68b0ff9700 1 -- 172.21.15.52:0/4176613030 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68a8013420 con 0x7f68b4101c40 2023-12-16T20:48:45.788 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.785+0000 7f68b0ff9700 1 -- 172.21.15.52:0/4176613030 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68a801c610 con 0x7f68b4101c40 2023-12-16T20:48:45.788 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.785+0000 7f68ba97a700 1 -- 172.21.15.52:0/4176613030 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b4117c30 con 0x7f68b4101c40 2023-12-16T20:48:45.788 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.786+0000 7f68ba97a700 1 -- 172.21.15.52:0/4176613030 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b4075170 con 0x7f68b4101c40 2023-12-16T20:48:45.788 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.786+0000 7f68b0ff9700 1 -- 172.21.15.52:0/4176613030 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f68a8013590 con 0x7f68b4101c40 2023-12-16T20:48:45.788 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.787+0000 7f68b0ff9700 1 --2- 172.21.15.52:0/4176613030 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f689c068f60 0x7f689c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:45.788 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.787+0000 7f68b0ff9700 1 -- 172.21.15.52:0/4176613030 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f68a8087cf0 con 0x7f68b4101c40 2023-12-16T20:48:45.789 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.787+0000 7f68b3fff700 1 --2- 172.21.15.52:0/4176613030 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f689c068f60 0x7f689c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:45.789 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.787+0000 7f68b3fff700 1 --2- 172.21.15.52:0/4176613030 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f689c068f60 0x7f689c06b410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f68a4006fd0 tx=0x7f68a4008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:45.793 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.791+0000 7f68b0ff9700 1 -- 172.21.15.52:0/4176613030 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f68a8054650 con 0x7f68b4101c40 2023-12-16T20:48:45.987 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.986+0000 7f68ba97a700 1 -- 172.21.15.52:0/4176613030 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68b4075170 con 0x7f68b4101c40 2023-12-16T20:48:45.987 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.987+0000 7f68b0ff9700 1 -- 172.21.15.52:0/4176613030 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+11078 (secure 0 0 0) 0x7f68a8057c70 con 0x7f68b4101c40 2023-12-16T20:48:45.988 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:48:45.988 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":37,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","created":"2023-12-16T20:44:44.679887+0000","modified":"2023-12-16T20:48:35.123510+0000","last_up_change":"2023-12-16T20:48:33.120930+0000","last_in_change":"2023-12-16T20:48:17.606842+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T20:47:46.398235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"03268674-f4fa-4c69-9fa5-ac2327873b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6803","nonce":362877775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6805","nonce":362877775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6809","nonce":362877775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6807","nonce":362877775}]},"public_addr":"172.21.15.52:6803/362877775","cluster_addr":"172.21.15.52:6805/362877775","heartbeat_back_addr":"172.21.15.52:6809/362877775","heartbeat_front_addr":"172.21.15.52:6807/362877775","state":["exists","up"]},{"osd":1,"uuid":"8cf65e84-7be8-4d6c-872d-d908e7a55c46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6811","nonce":2532880721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6813","nonce":2532880721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6817","nonce":2532880721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6815","nonce":2532880721}]},"public_addr":"172.21.15.52:6811/2532880721","cluster_addr":"172.21.15.52:6813/2532880721","heartbeat_back_addr":"172.21.15.52:6817/2532880721","heartbeat_front_addr":"172.21.15.52:6815/2532880721","state":["exists","up"]},{"osd":2,"uuid":"fd375c62-57d0-4899-aa93-b582a664abbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6819","nonce":4281298930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6821","nonce":4281298930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6825","nonce":4281298930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6823","nonce":4281298930}]},"public_addr":"172.21.15.52:6819/4281298930","cluster_addr":"172.21.15.52:6821/4281298930","heartbeat_back_addr":"172.21.15.52:6825/4281298930","heartbeat_front_addr":"172.21.15.52:6823/4281298930","state":["exists","up"]},{"osd":3,"uuid":"31298367-7548-4ef5-bd2b-a6f586ec14a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6801","nonce":2861125970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6803","nonce":2861125970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6807","nonce":2861125970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6805","nonce":2861125970}]},"public_addr":"172.21.15.150:6801/2861125970","cluster_addr":"172.21.15.150:6803/2861125970","heartbeat_back_addr":"172.21.15.150:6807/2861125970","heartbeat_front_addr":"172.21.15.150:6805/2861125970","state":["exists","up"]},{"osd":4,"uuid":"ab02c027-f21c-4187-af78-f4daaa844db1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6809","nonce":2997946868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6811","nonce":2997946868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6815","nonce":2997946868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6813","nonce":2997946868}]},"public_addr":"172.21.15.150:6809/2997946868","cluster_addr":"172.21.15.150:6811/2997946868","heartbeat_back_addr":"172.21.15.150:6815/2997946868","heartbeat_front_addr":"172.21.15.150:6813/2997946868","state":["exists","up"]},{"osd":5,"uuid":"f62123bf-fcd1-4253-b8c7-78c2379bb90b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6817","nonce":4046379778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6819","nonce":4046379778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6823","nonce":4046379778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6821","nonce":4046379778}]},"public_addr":"172.21.15.150:6817/4046379778","cluster_addr":"172.21.15.150:6819/4046379778","heartbeat_back_addr":"172.21.15.150:6823/4046379778","heartbeat_front_addr":"172.21.15.150:6821/4046379778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:06.709733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:25.479953+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:44.608913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:59.806350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:48:15.471651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:48:31.147029+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6800/2799692744":"2023-12-17T20:46:17.883785+0000","172.21.15.52:0/4160218759":"2023-12-17T20:46:17.883785+0000","172.21.15.52:0/2518812785":"2023-12-17T20:46:17.883785+0000","172.21.15.52:6800/221719009":"2023-12-17T20:45:28.570795+0000","172.21.15.52:0/2730302642":"2023-12-17T20:45:28.570795+0000","172.21.15.52:6801/2799692744":"2023-12-17T20:46:17.883785+0000","172.21.15.52:0/2975128681":"2023-12-17T20:45:28.570795+0000","172.21.15.52:0/806767676":"2023-12-17T20:45:07.199601+0000","172.21.15.52:0/82552375":"2023-12-17T20:45:07.199601+0000","172.21.15.52:0/1810720520":"2023-12-17T20:45:28.570795+0000","172.21.15.52:0/1900058090":"2023-12-17T20:46:17.883785+0000","172.21.15.52:6801/221719009":"2023-12-17T20:45:28.570795+0000","172.21.15.52:6800/2740978054":"2023-12-17T20:45:07.199601+0000","172.21.15.52:6801/2740978054":"2023-12-17T20:45:07.199601+0000","172.21.15.52:0/27878571":"2023-12-17T20:45:07.199601+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T20:48:45.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.989+0000 7f68ba97a700 1 -- 172.21.15.52:0/4176613030 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f689c068f60 msgr2=0x7f689c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:45.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.989+0000 7f68ba97a700 1 --2- 172.21.15.52:0/4176613030 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f689c068f60 0x7f689c06b410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f68a4006fd0 tx=0x7f68a4008040 comp rx=0 tx=0).stop 2023-12-16T20:48:45.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.989+0000 7f68ba97a700 1 -- 172.21.15.52:0/4176613030 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68b4101c40 msgr2=0x7f68b410ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:45.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.989+0000 7f68ba97a700 1 --2- 172.21.15.52:0/4176613030 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68b4101c40 0x7f68b410ecc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009100 tx=0x7f68a8009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:45.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.989+0000 7f68ba97a700 1 -- 172.21.15.52:0/4176613030 shutdown_connections 2023-12-16T20:48:45.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.989+0000 7f68ba97a700 1 --2- 172.21.15.52:0/4176613030 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f689c068f60 0x7f689c06b410 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:45.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.989+0000 7f68ba97a700 1 --2- 172.21.15.52:0/4176613030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68b40fc320 0x7f68b410e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:45.991 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.989+0000 7f68ba97a700 1 --2- 172.21.15.52:0/4176613030 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68b4101c40 0x7f68b410ecc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:45.991 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.989+0000 7f68ba97a700 1 -- 172.21.15.52:0/4176613030 >> 172.21.15.52:0/4176613030 conn(0x7f68b40711b0 msgr2=0x7f68b40f9360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:45.991 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.990+0000 7f68ba97a700 1 -- 172.21.15.52:0/4176613030 shutdown_connections 2023-12-16T20:48:45.991 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:45.990+0000 7f68ba97a700 1 -- 172.21.15.52:0/4176613030 wait complete. 2023-12-16T20:48:46.334 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-12-16T20:48:46.334 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-16T20:48:46.598 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:47.216 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:46 smithi052 ceph-mon[80761]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:47.217 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:46 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/4176613030' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T20:48:47.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:46 smithi150 ceph-mon[88000]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:47.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:46 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/4176613030' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T20:48:47.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.440+0000 7f5693b8c700 1 -- 172.21.15.52:0/2200209316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f568c102c00 msgr2=0x7f568c102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:47.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.440+0000 7f5693b8c700 1 --2- 172.21.15.52:0/2200209316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f568c102c00 0x7f568c102fd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5684009ce0 tx=0x7f5684004d10 comp rx=0 tx=0).stop 2023-12-16T20:48:47.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.440+0000 7f5693b8c700 1 -- 172.21.15.52:0/2200209316 shutdown_connections 2023-12-16T20:48:47.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.440+0000 7f5693b8c700 1 --2- 172.21.15.52:0/2200209316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c1035a0 0x7f568c0fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:47.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.440+0000 7f5693b8c700 1 --2- 172.21.15.52:0/2200209316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f568c102c00 0x7f568c102fd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:47.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.440+0000 7f5693b8c700 1 -- 172.21.15.52:0/2200209316 >> 172.21.15.52:0/2200209316 conn(0x7f568c0f8790 msgr2=0x7f568c0faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:47.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.440+0000 7f5693b8c700 1 -- 172.21.15.52:0/2200209316 shutdown_connections 2023-12-16T20:48:47.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.441+0000 7f5693b8c700 1 -- 172.21.15.52:0/2200209316 wait complete. 2023-12-16T20:48:47.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.441+0000 7f5693b8c700 1 Processor -- start 2023-12-16T20:48:47.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.441+0000 7f5693b8c700 1 -- start start 2023-12-16T20:48:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.441+0000 7f5693b8c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c102c00 0x7f568c06ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.441+0000 7f5693b8c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f568c1035a0 0x7f568c070480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.441+0000 7f5693b8c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c070a80 con 0x7f568c1035a0 2023-12-16T20:48:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.441+0000 7f5693b8c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c070bf0 con 0x7f568c102c00 2023-12-16T20:48:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.441+0000 7f5691127700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f568c1035a0 0x7f568c070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.441+0000 7f5691127700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f568c1035a0 0x7f568c070480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38202/0 (socket says 172.21.15.52:38202) 2023-12-16T20:48:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.441+0000 7f5691127700 1 -- 172.21.15.52:0/3819544153 learned_addr learned my addr 172.21.15.52:0/3819544153 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.442+0000 7f5691928700 1 --2- 172.21.15.52:0/3819544153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c102c00 0x7f568c06ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.442+0000 7f5691127700 1 -- 172.21.15.52:0/3819544153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c102c00 msgr2=0x7f568c06ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.442+0000 7f5691127700 1 --2- 172.21.15.52:0/3819544153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c102c00 0x7f568c06ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.442+0000 7f5691127700 1 -- 172.21.15.52:0/3819544153 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5684009940 con 0x7f568c1035a0 2023-12-16T20:48:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.442+0000 7f5691127700 1 --2- 172.21.15.52:0/3819544153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f568c1035a0 0x7f568c070480 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5680002a50 tx=0x7f5680004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.442+0000 7f568a7fc700 1 -- 172.21.15.52:0/3819544153 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5680007cc0 con 0x7f568c1035a0 2023-12-16T20:48:47.445 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.442+0000 7f568a7fc700 1 -- 172.21.15.52:0/3819544153 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f568000f040 con 0x7f568c1035a0 2023-12-16T20:48:47.445 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.442+0000 7f5693b8c700 1 -- 172.21.15.52:0/3819544153 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568c1199d0 con 0x7f568c1035a0 2023-12-16T20:48:47.445 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.442+0000 7f568a7fc700 1 -- 172.21.15.52:0/3819544153 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f568000d920 con 0x7f568c1035a0 2023-12-16T20:48:47.445 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.442+0000 7f5693b8c700 1 -- 172.21.15.52:0/3819544153 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568c119eb0 con 0x7f568c1035a0 2023-12-16T20:48:47.446 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.443+0000 7f5693b8c700 1 -- 172.21.15.52:0/3819544153 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f568c0fff10 con 0x7f568c1035a0 2023-12-16T20:48:47.446 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.444+0000 7f568a7fc700 1 -- 172.21.15.52:0/3819544153 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f5680009680 con 0x7f568c1035a0 2023-12-16T20:48:47.446 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.444+0000 7f568a7fc700 1 --2- 172.21.15.52:0/3819544153 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5674068f60 0x7f567406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:47.446 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.444+0000 7f568a7fc700 1 -- 172.21.15.52:0/3819544153 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f56800867f0 con 0x7f568c1035a0 2023-12-16T20:48:47.446 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.444+0000 7f5691928700 1 --2- 172.21.15.52:0/3819544153 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5674068f60 0x7f567406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:47.446 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.445+0000 7f5691928700 1 --2- 172.21.15.52:0/3819544153 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5674068f60 0x7f567406b410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5684003400 tx=0x7f5684004d10 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:47.449 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.449+0000 7f568a7fc700 1 -- 172.21.15.52:0/3819544153 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5680052f80 con 0x7f568c1035a0 2023-12-16T20:48:47.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.644+0000 7f5693b8c700 1 -- 172.21.15.52:0/3819544153 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f568c0fc390 con 0x7f568c1035a0 2023-12-16T20:48:47.646 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.645+0000 7f568a7fc700 1 -- 172.21.15.52:0/3819544153 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+11078 (secure 0 0 0) 0x7f56800565a0 con 0x7f568c1035a0 2023-12-16T20:48:47.647 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:48:47.647 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":37,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","created":"2023-12-16T20:44:44.679887+0000","modified":"2023-12-16T20:48:35.123510+0000","last_up_change":"2023-12-16T20:48:33.120930+0000","last_in_change":"2023-12-16T20:48:17.606842+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T20:47:46.398235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"03268674-f4fa-4c69-9fa5-ac2327873b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6803","nonce":362877775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6805","nonce":362877775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6809","nonce":362877775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6807","nonce":362877775}]},"public_addr":"172.21.15.52:6803/362877775","cluster_addr":"172.21.15.52:6805/362877775","heartbeat_back_addr":"172.21.15.52:6809/362877775","heartbeat_front_addr":"172.21.15.52:6807/362877775","state":["exists","up"]},{"osd":1,"uuid":"8cf65e84-7be8-4d6c-872d-d908e7a55c46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6811","nonce":2532880721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6813","nonce":2532880721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6817","nonce":2532880721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6815","nonce":2532880721}]},"public_addr":"172.21.15.52:6811/2532880721","cluster_addr":"172.21.15.52:6813/2532880721","heartbeat_back_addr":"172.21.15.52:6817/2532880721","heartbeat_front_addr":"172.21.15.52:6815/2532880721","state":["exists","up"]},{"osd":2,"uuid":"fd375c62-57d0-4899-aa93-b582a664abbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6819","nonce":4281298930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6821","nonce":4281298930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6825","nonce":4281298930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6823","nonce":4281298930}]},"public_addr":"172.21.15.52:6819/4281298930","cluster_addr":"172.21.15.52:6821/4281298930","heartbeat_back_addr":"172.21.15.52:6825/4281298930","heartbeat_front_addr":"172.21.15.52:6823/4281298930","state":["exists","up"]},{"osd":3,"uuid":"31298367-7548-4ef5-bd2b-a6f586ec14a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6801","nonce":2861125970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6803","nonce":2861125970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6807","nonce":2861125970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6805","nonce":2861125970}]},"public_addr":"172.21.15.150:6801/2861125970","cluster_addr":"172.21.15.150:6803/2861125970","heartbeat_back_addr":"172.21.15.150:6807/2861125970","heartbeat_front_addr":"172.21.15.150:6805/2861125970","state":["exists","up"]},{"osd":4,"uuid":"ab02c027-f21c-4187-af78-f4daaa844db1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6809","nonce":2997946868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6811","nonce":2997946868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6815","nonce":2997946868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6813","nonce":2997946868}]},"public_addr":"172.21.15.150:6809/2997946868","cluster_addr":"172.21.15.150:6811/2997946868","heartbeat_back_addr":"172.21.15.150:6815/2997946868","heartbeat_front_addr":"172.21.15.150:6813/2997946868","state":["exists","up"]},{"osd":5,"uuid":"f62123bf-fcd1-4253-b8c7-78c2379bb90b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6817","nonce":4046379778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6819","nonce":4046379778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6823","nonce":4046379778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6821","nonce":4046379778}]},"public_addr":"172.21.15.150:6817/4046379778","cluster_addr":"172.21.15.150:6819/4046379778","heartbeat_back_addr":"172.21.15.150:6823/4046379778","heartbeat_front_addr":"172.21.15.150:6821/4046379778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:06.709733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:25.479953+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:44.608913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:59.806350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:48:15.471651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:48:31.147029+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6800/2799692744":"2023-12-17T20:46:17.883785+0000","172.21.15.52:0/4160218759":"2023-12-17T20:46:17.883785+0000","172.21.15.52:0/2518812785":"2023-12-17T20:46:17.883785+0000","172.21.15.52:6800/221719009":"2023-12-17T20:45:28.570795+0000","172.21.15.52:0/2730302642":"2023-12-17T20:45:28.570795+0000","172.21.15.52:6801/2799692744":"2023-12-17T20:46:17.883785+0000","172.21.15.52:0/2975128681":"2023-12-17T20:45:28.570795+0000","172.21.15.52:0/806767676":"2023-12-17T20:45:07.199601+0000","172.21.15.52:0/82552375":"2023-12-17T20:45:07.199601+0000","172.21.15.52:0/1810720520":"2023-12-17T20:45:28.570795+0000","172.21.15.52:0/1900058090":"2023-12-17T20:46:17.883785+0000","172.21.15.52:6801/221719009":"2023-12-17T20:45:28.570795+0000","172.21.15.52:6800/2740978054":"2023-12-17T20:45:07.199601+0000","172.21.15.52:6801/2740978054":"2023-12-17T20:45:07.199601+0000","172.21.15.52:0/27878571":"2023-12-17T20:45:07.199601+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T20:48:47.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.647+0000 7f5693b8c700 1 -- 172.21.15.52:0/3819544153 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5674068f60 msgr2=0x7f567406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:47.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.647+0000 7f5693b8c700 1 --2- 172.21.15.52:0/3819544153 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5674068f60 0x7f567406b410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5684003400 tx=0x7f5684004d10 comp rx=0 tx=0).stop 2023-12-16T20:48:47.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.647+0000 7f5693b8c700 1 -- 172.21.15.52:0/3819544153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f568c1035a0 msgr2=0x7f568c070480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:47.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.647+0000 7f5693b8c700 1 --2- 172.21.15.52:0/3819544153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f568c1035a0 0x7f568c070480 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5680002a50 tx=0x7f5680004600 comp rx=0 tx=0).stop 2023-12-16T20:48:47.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.647+0000 7f5693b8c700 1 -- 172.21.15.52:0/3819544153 shutdown_connections 2023-12-16T20:48:47.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.647+0000 7f5693b8c700 1 --2- 172.21.15.52:0/3819544153 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5674068f60 0x7f567406b410 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:47.649 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.647+0000 7f5693b8c700 1 --2- 172.21.15.52:0/3819544153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c102c00 0x7f568c06ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:47.649 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.647+0000 7f5693b8c700 1 --2- 172.21.15.52:0/3819544153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f568c1035a0 0x7f568c070480 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:47.649 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.647+0000 7f5693b8c700 1 -- 172.21.15.52:0/3819544153 >> 172.21.15.52:0/3819544153 conn(0x7f568c0f8790 msgr2=0x7f568c0fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:47.649 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.648+0000 7f5693b8c700 1 -- 172.21.15.52:0/3819544153 shutdown_connections 2023-12-16T20:48:47.649 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:47.648+0000 7f5693b8c700 1 -- 172.21.15.52:0/3819544153 wait complete. 2023-12-16T20:48:48.018 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2023-12-16T20:48:48.018 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2023-12-16T20:48:48.019 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2023-12-16T20:48:48.019 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2023-12-16T20:48:48.019 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2023-12-16T20:48:48.020 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2023-12-16T20:48:48.342 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:48.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:48 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/3819544153' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T20:48:48.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:48 smithi150 ceph-mon[88000]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:48.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:48 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:48:48.708 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:48 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3819544153' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T20:48:48.708 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:48 smithi052 ceph-mon[80761]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:48.708 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:48 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:48:48.720 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:49.681 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:50.011 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.010+0000 7f573ea78700 1 -- 172.21.15.52:0/2225540688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5738101b60 msgr2=0x7f5738101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:50.011 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.010+0000 7f573ea78700 1 --2- 172.21.15.52:0/2225540688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5738101b60 0x7f5738101f30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5730009ce0 tx=0x7f5730004d10 comp rx=0 tx=0).stop 2023-12-16T20:48:50.011 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.010+0000 7f573ea78700 1 -- 172.21.15.52:0/2225540688 shutdown_connections 2023-12-16T20:48:50.011 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.010+0000 7f573ea78700 1 --2- 172.21.15.52:0/2225540688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57380fbb00 0x7f57380fbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.012 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.010+0000 7f573ea78700 1 --2- 172.21.15.52:0/2225540688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5738101b60 0x7f5738101f30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.012 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.010+0000 7f573ea78700 1 -- 172.21.15.52:0/2225540688 >> 172.21.15.52:0/2225540688 conn(0x7f57380f7680 msgr2=0x7f57380f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:50.012 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.010+0000 7f573ea78700 1 -- 172.21.15.52:0/2225540688 shutdown_connections 2023-12-16T20:48:50.012 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.010+0000 7f573ea78700 1 -- 172.21.15.52:0/2225540688 wait complete. 2023-12-16T20:48:50.012 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.011+0000 7f573ea78700 1 Processor -- start 2023-12-16T20:48:50.012 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.011+0000 7f573ea78700 1 -- start start 2023-12-16T20:48:50.013 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.011+0000 7f573ea78700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57380fbb00 0x7f57380fd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:50.013 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.011+0000 7f573ea78700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5738101b60 0x7f57380fdc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:50.014 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.011+0000 7f573ea78700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57380fe210 con 0x7f5738101b60 2023-12-16T20:48:50.014 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.011+0000 7f573ea78700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57380fe380 con 0x7f57380fbb00 2023-12-16T20:48:50.014 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.011+0000 7f5737fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5738101b60 0x7f57380fdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:50.014 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.011+0000 7f5737fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5738101b60 0x7f57380fdc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38226/0 (socket says 172.21.15.52:38226) 2023-12-16T20:48:50.014 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.011+0000 7f5737fff700 1 -- 172.21.15.52:0/1364504894 learned_addr learned my addr 172.21.15.52:0/1364504894 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:50.014 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.011+0000 7f573c814700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57380fbb00 0x7f57380fd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:50.014 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.011+0000 7f5737fff700 1 -- 172.21.15.52:0/1364504894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57380fbb00 msgr2=0x7f57380fd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:50.014 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.011+0000 7f5737fff700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57380fbb00 0x7f57380fd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.014 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.011+0000 7f5737fff700 1 -- 172.21.15.52:0/1364504894 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5730009940 con 0x7f5738101b60 2023-12-16T20:48:50.014 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.012+0000 7f5737fff700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5738101b60 0x7f57380fdc10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f572c002a50 tx=0x7f572c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:50.014 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.012+0000 7f57357fa700 1 -- 172.21.15.52:0/1364504894 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f572c007cc0 con 0x7f5738101b60 2023-12-16T20:48:50.014 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.012+0000 7f57357fa700 1 -- 172.21.15.52:0/1364504894 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f572c00f040 con 0x7f5738101b60 2023-12-16T20:48:50.014 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.012+0000 7f573ea78700 1 -- 172.21.15.52:0/1364504894 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57381199f0 con 0x7f5738101b60 2023-12-16T20:48:50.014 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.012+0000 7f57357fa700 1 -- 172.21.15.52:0/1364504894 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f572c00da70 con 0x7f5738101b60 2023-12-16T20:48:50.015 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.012+0000 7f573ea78700 1 -- 172.21.15.52:0/1364504894 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5738119ed0 con 0x7f5738101b60 2023-12-16T20:48:50.015 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.013+0000 7f573ea78700 1 -- 172.21.15.52:0/1364504894 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f573805e380 con 0x7f5738101b60 2023-12-16T20:48:50.015 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.013+0000 7f57357fa700 1 -- 172.21.15.52:0/1364504894 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f572c00dc10 con 0x7f5738101b60 2023-12-16T20:48:50.016 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.014+0000 7f57357fa700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5720068f60 0x7f572006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:50.016 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.014+0000 7f573c814700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5720068f60 0x7f572006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:50.016 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.014+0000 7f57357fa700 1 -- 172.21.15.52:0/1364504894 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f572c0861b0 con 0x7f5738101b60 2023-12-16T20:48:50.016 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.014+0000 7f57357fa700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775] conn(0x7f572006ec30 0x7f5720071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:50.016 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.014+0000 7f57357fa700 1 -- 172.21.15.52:0/1364504894 --> [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57200716f0 con 0x7f572006ec30 2023-12-16T20:48:50.016 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.014+0000 7f57357fa700 1 -- 172.21.15.52:0/1364504894 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f572c0887a0 con 0x7f5738101b60 2023-12-16T20:48:50.016 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.014+0000 7f573d015700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775] conn(0x7f572006ec30 0x7f5720071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:50.016 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.015+0000 7f573c814700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5720068f60 0x7f572006b410 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f573000b740 tx=0x7f5730004d10 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:50.016 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.015+0000 7f573d015700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775] conn(0x7f572006ec30 0x7f5720071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:50.016 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.016+0000 7f57357fa700 1 -- 172.21.15.52:0/1364504894 <== osd.0 v2:172.21.15.52:6802/362877775 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f57200716f0 con 0x7f572006ec30 2023-12-16T20:48:50.032 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.031+0000 7f573ea78700 1 -- 172.21.15.52:0/1364504894 --> [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f573805e830 con 0x7f572006ec30 2023-12-16T20:48:50.032 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.031+0000 7f57357fa700 1 -- 172.21.15.52:0/1364504894 <== osd.0 v2:172.21.15.52:6802/362877775 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f573805e830 con 0x7f572006ec30 2023-12-16T20:48:50.033 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.032+0000 7f573ea78700 1 -- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775] conn(0x7f572006ec30 msgr2=0x7f5720071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:50.033 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.032+0000 7f573ea78700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775] conn(0x7f572006ec30 0x7f5720071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.033 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.032+0000 7f573ea78700 1 -- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5720068f60 msgr2=0x7f572006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:50.033 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.032+0000 7f573ea78700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5720068f60 0x7f572006b410 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f573000b740 tx=0x7f5730004d10 comp rx=0 tx=0).stop 2023-12-16T20:48:50.033 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.032+0000 7f573ea78700 1 -- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5738101b60 msgr2=0x7f57380fdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:50.033 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.032+0000 7f573ea78700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5738101b60 0x7f57380fdc10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f572c002a50 tx=0x7f572c0046b0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.033 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.032+0000 7f573ea78700 1 -- 172.21.15.52:0/1364504894 shutdown_connections 2023-12-16T20:48:50.033 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.032+0000 7f573ea78700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:6802/362877775,v1:172.21.15.52:6803/362877775] conn(0x7f572006ec30 0x7f5720071040 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.034 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.032+0000 7f573ea78700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5720068f60 0x7f572006b410 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.034 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.032+0000 7f573ea78700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57380fbb00 0x7f57380fd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.034 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.032+0000 7f573ea78700 1 --2- 172.21.15.52:0/1364504894 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5738101b60 0x7f57380fdc10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.034 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.032+0000 7f573ea78700 1 -- 172.21.15.52:0/1364504894 >> 172.21.15.52:0/1364504894 conn(0x7f57380f7680 msgr2=0x7f5738065f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:50.034 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.032+0000 7f573ea78700 1 -- 172.21.15.52:0/1364504894 shutdown_connections 2023-12-16T20:48:50.034 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.032+0000 7f573ea78700 1 -- 172.21.15.52:0/1364504894 wait complete. 2023-12-16T20:48:50.333 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:50.434 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:50.435 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:50.605 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.601+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1607571827 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d801035a0 msgr2=0x7f7d800fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:50.605 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.601+0000 7f7d85ecc700 1 --2- 172.21.15.52:0/1607571827 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d801035a0 0x7f7d800fcc10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7d74004940 tx=0x7f7d74009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:50.605 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.601+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1607571827 shutdown_connections 2023-12-16T20:48:50.605 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.601+0000 7f7d85ecc700 1 --2- 172.21.15.52:0/1607571827 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d801035a0 0x7f7d800fcc10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.605 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.601+0000 7f7d85ecc700 1 --2- 172.21.15.52:0/1607571827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d80102c00 0x7f7d80102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.605 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.601+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1607571827 >> 172.21.15.52:0/1607571827 conn(0x7f7d800f8790 msgr2=0x7f7d800faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:50.606 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.601+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1607571827 shutdown_connections 2023-12-16T20:48:50.606 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.601+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1607571827 wait complete. 2023-12-16T20:48:50.606 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.602+0000 7f7d85ecc700 1 Processor -- start 2023-12-16T20:48:50.606 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.602+0000 7f7d85ecc700 1 -- start start 2023-12-16T20:48:50.606 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.602+0000 7f7d85ecc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d80102c00 0x7f7d80113680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:50.606 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.602+0000 7f7d85ecc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d801035a0 0x7f7d80113bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:50.606 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.602+0000 7f7d85ecc700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d80114210 con 0x7f7d80102c00 2023-12-16T20:48:50.607 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.602+0000 7f7d85ecc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8010d500 con 0x7f7d801035a0 2023-12-16T20:48:50.607 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.602+0000 7f7d7f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d80102c00 0x7f7d80113680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:50.607 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.602+0000 7f7d7f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d80102c00 0x7f7d80113680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38246/0 (socket says 172.21.15.52:38246) 2023-12-16T20:48:50.607 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.602+0000 7f7d7f7fe700 1 -- 172.21.15.52:0/1532269756 learned_addr learned my addr 172.21.15.52:0/1532269756 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:50.607 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.602+0000 7f7d7effd700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d801035a0 0x7f7d80113bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:50.607 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.602+0000 7f7d7f7fe700 1 -- 172.21.15.52:0/1532269756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d801035a0 msgr2=0x7f7d80113bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:50.607 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.602+0000 7f7d7f7fe700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d801035a0 0x7f7d80113bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.607 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.603+0000 7f7d7f7fe700 1 -- 172.21.15.52:0/1532269756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d74005040 con 0x7f7d80102c00 2023-12-16T20:48:50.607 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.603+0000 7f7d7f7fe700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d80102c00 0x7f7d80113680 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000c8c0 tx=0x7f7d700044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:50.607 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.603+0000 7f7d67fff700 1 -- 172.21.15.52:0/1532269756 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d7000a760 con 0x7f7d80102c00 2023-12-16T20:48:50.607 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.603+0000 7f7d67fff700 1 -- 172.21.15.52:0/1532269756 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d7000ad70 con 0x7f7d80102c00 2023-12-16T20:48:50.607 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.603+0000 7f7d67fff700 1 -- 172.21.15.52:0/1532269756 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d700113a0 con 0x7f7d80102c00 2023-12-16T20:48:50.607 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.603+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1532269756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8010d7e0 con 0x7f7d80102c00 2023-12-16T20:48:50.607 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.603+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1532269756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8010dd20 con 0x7f7d80102c00 2023-12-16T20:48:50.608 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.604+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1532269756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7d8010e2d0 con 0x7f7d80102c00 2023-12-16T20:48:50.608 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.604+0000 7f7d67fff700 1 -- 172.21.15.52:0/1532269756 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f7d70012400 con 0x7f7d80102c00 2023-12-16T20:48:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.605+0000 7f7d67fff700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7d68068f60 0x7f7d6806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.605+0000 7f7d67fff700 1 -- 172.21.15.52:0/1532269756 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7d70086a90 con 0x7f7d80102c00 2023-12-16T20:48:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.605+0000 7f7d7effd700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7d68068f60 0x7f7d6806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.605+0000 7f7d67fff700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721] conn(0x7f7d6806ec30 0x7f7d68071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.605+0000 7f7d67fff700 1 -- 172.21.15.52:0/1532269756 --> [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d680716f0 con 0x7f7d6806ec30 2023-12-16T20:48:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.605+0000 7f7d67fff700 1 -- 172.21.15.52:0/1532269756 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d7008b220 con 0x7f7d80102c00 2023-12-16T20:48:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.605+0000 7f7d7ffff700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721] conn(0x7f7d6806ec30 0x7f7d68071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.605+0000 7f7d7effd700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7d68068f60 0x7f7d6806b410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7d8010eee0 tx=0x7f7d74009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.606+0000 7f7d7ffff700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721] conn(0x7f7d6806ec30 0x7f7d68071040 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.606+0000 7f7d67fff700 1 -- 172.21.15.52:0/1532269756 <== osd.1 v2:172.21.15.52:6810/2532880721 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f7d680716f0 con 0x7f7d6806ec30 2023-12-16T20:48:50.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.621+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1532269756 --> [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7d8005e340 con 0x7f7d6806ec30 2023-12-16T20:48:50.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.622+0000 7f7d67fff700 1 -- 172.21.15.52:0/1532269756 <== osd.1 v2:172.21.15.52:6810/2532880721 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7d8005e340 con 0x7f7d6806ec30 2023-12-16T20:48:50.623 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.622+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721] conn(0x7f7d6806ec30 msgr2=0x7f7d68071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:50.623 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.622+0000 7f7d85ecc700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721] conn(0x7f7d6806ec30 0x7f7d68071040 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.623 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.622+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7d68068f60 msgr2=0x7f7d6806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:50.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.622+0000 7f7d85ecc700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7d68068f60 0x7f7d6806b410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7d8010eee0 tx=0x7f7d74009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:50.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.622+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d80102c00 msgr2=0x7f7d80113680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:50.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.622+0000 7f7d85ecc700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d80102c00 0x7f7d80113680 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000c8c0 tx=0x7f7d700044c0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.622+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1532269756 shutdown_connections 2023-12-16T20:48:50.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.622+0000 7f7d85ecc700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:6810/2532880721,v1:172.21.15.52:6811/2532880721] conn(0x7f7d6806ec30 0x7f7d68071040 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.622+0000 7f7d85ecc700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7d68068f60 0x7f7d6806b410 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.622+0000 7f7d85ecc700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d80102c00 0x7f7d80113680 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.622+0000 7f7d85ecc700 1 --2- 172.21.15.52:0/1532269756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d801035a0 0x7f7d80113bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:50.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.622+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1532269756 >> 172.21.15.52:0/1532269756 conn(0x7f7d800f8790 msgr2=0x7f7d800fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:50.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.622+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1532269756 shutdown_connections 2023-12-16T20:48:50.624 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:50.622+0000 7f7d85ecc700 1 -- 172.21.15.52:0/1532269756 wait complete. 2023-12-16T20:48:50.844 INFO:teuthology.orchestra.run.smithi052.stdout:38654705687 2023-12-16T20:48:50.844 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.0 2023-12-16T20:48:51.141 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:50 smithi052 ceph-mon[80761]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:51.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:50 smithi150 ceph-mon[88000]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:53.003 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.002+0000 7fcb8e824700 1 -- 172.21.15.52:0/2140531846 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb88102c70 msgr2=0x7fcb88103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:53.003 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.002+0000 7fcb8e824700 1 --2- 172.21.15.52:0/2140531846 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb88102c70 0x7fcb88103040 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcb78004940 tx=0x7fcb78009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:53.003 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.002+0000 7fcb8e824700 1 -- 172.21.15.52:0/2140531846 shutdown_connections 2023-12-16T20:48:53.003 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.002+0000 7fcb8e824700 1 --2- 172.21.15.52:0/2140531846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb880fb1e0 0x7fcb880fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.003 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.002+0000 7fcb8e824700 1 --2- 172.21.15.52:0/2140531846 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb88102c70 0x7fcb88103040 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.003 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.002+0000 7fcb8e824700 1 -- 172.21.15.52:0/2140531846 >> 172.21.15.52:0/2140531846 conn(0x7fcb880711a0 msgr2=0x7fcb880715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:53.003 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.002+0000 7fcb8e824700 1 -- 172.21.15.52:0/2140531846 shutdown_connections 2023-12-16T20:48:53.004 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.002+0000 7fcb8e824700 1 -- 172.21.15.52:0/2140531846 wait complete. 2023-12-16T20:48:53.004 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.002+0000 7fcb8e824700 1 Processor -- start 2023-12-16T20:48:53.004 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.003+0000 7fcb8e824700 1 -- start start 2023-12-16T20:48:53.005 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.003+0000 7fcb8e824700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb880fb1e0 0x7fcb880fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:53.005 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.003+0000 7fcb8e824700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb88102c70 0x7fcb880fc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:53.005 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.003+0000 7fcb8e824700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb880fcf90 con 0x7fcb880fb1e0 2023-12-16T20:48:53.005 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.003+0000 7fcb8e824700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb880fd100 con 0x7fcb88102c70 2023-12-16T20:48:53.005 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.003+0000 7fcb87fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb880fb1e0 0x7fcb880fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:53.005 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.003+0000 7fcb87fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb880fb1e0 0x7fcb880fc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35422/0 (socket says 172.21.15.52:35422) 2023-12-16T20:48:53.005 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.003+0000 7fcb87fff700 1 -- 172.21.15.52:0/3709669600 learned_addr learned my addr 172.21.15.52:0/3709669600 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:53.006 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.003+0000 7fcb87fff700 1 -- 172.21.15.52:0/3709669600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb88102c70 msgr2=0x7fcb880fc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:53.006 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.003+0000 7fcb87fff700 1 --2- 172.21.15.52:0/3709669600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb88102c70 0x7fcb880fc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.006 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.003+0000 7fcb87fff700 1 -- 172.21.15.52:0/3709669600 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb78005040 con 0x7fcb880fb1e0 2023-12-16T20:48:53.006 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.003+0000 7fcb87fff700 1 --2- 172.21.15.52:0/3709669600 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb880fb1e0 0x7fcb880fc3c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009180 tx=0x7fcb78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:53.006 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.003+0000 7fcb84ff9700 1 -- 172.21.15.52:0/3709669600 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb7800e040 con 0x7fcb880fb1e0 2023-12-16T20:48:53.006 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.004+0000 7fcb84ff9700 1 -- 172.21.15.52:0/3709669600 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb78013420 con 0x7fcb880fb1e0 2023-12-16T20:48:53.006 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.004+0000 7fcb84ff9700 1 -- 172.21.15.52:0/3709669600 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb7801c630 con 0x7fcb880fb1e0 2023-12-16T20:48:53.006 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.004+0000 7fcb8e824700 1 -- 172.21.15.52:0/3709669600 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb88119830 con 0x7fcb880fb1e0 2023-12-16T20:48:53.006 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.004+0000 7fcb8e824700 1 -- 172.21.15.52:0/3709669600 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb88119d20 con 0x7fcb880fb1e0 2023-12-16T20:48:53.006 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.005+0000 7fcb8e824700 1 -- 172.21.15.52:0/3709669600 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcb880faa40 con 0x7fcb880fb1e0 2023-12-16T20:48:53.007 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.006+0000 7fcb84ff9700 1 -- 172.21.15.52:0/3709669600 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fcb780139e0 con 0x7fcb880fb1e0 2023-12-16T20:48:53.007 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.006+0000 7fcb84ff9700 1 --2- 172.21.15.52:0/3709669600 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fcb70068f60 0x7fcb7006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:53.007 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.006+0000 7fcb84ff9700 1 -- 172.21.15.52:0/3709669600 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcb78088800 con 0x7fcb880fb1e0 2023-12-16T20:48:53.007 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.006+0000 7fcb877fe700 1 --2- 172.21.15.52:0/3709669600 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fcb70068f60 0x7fcb7006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:53.007 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.006+0000 7fcb84ff9700 1 --2- 172.21.15.52:0/3709669600 >> [v2:172.21.15.150:6808/2997946868,v1:172.21.15.150:6809/2997946868] conn(0x7fcb7006ec30 0x7fcb70071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:53.007 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.006+0000 7fcb84ff9700 1 -- 172.21.15.52:0/3709669600 --> [v2:172.21.15.150:6808/2997946868,v1:172.21.15.150:6809/2997946868] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb700716f0 con 0x7fcb7006ec30 2023-12-16T20:48:53.008 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.006+0000 7fcb84ff9700 1 -- 172.21.15.52:0/3709669600 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb78088be0 con 0x7fcb880fb1e0 2023-12-16T20:48:53.008 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.007+0000 7fcb877fe700 1 --2- 172.21.15.52:0/3709669600 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fcb70068f60 0x7fcb7006b410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c006fd0 tx=0x7fcb7c006db0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:53.008 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.007+0000 7fcb8cdc1700 1 --2- 172.21.15.52:0/3709669600 >> [v2:172.21.15.150:6808/2997946868,v1:172.21.15.150:6809/2997946868] conn(0x7fcb7006ec30 0x7fcb70071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:53.008 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.007+0000 7fcb8cdc1700 1 --2- 172.21.15.52:0/3709669600 >> [v2:172.21.15.150:6808/2997946868,v1:172.21.15.150:6809/2997946868] conn(0x7fcb7006ec30 0x7fcb70071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:53.008 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.008+0000 7fcb84ff9700 1 -- 172.21.15.52:0/3709669600 <== osd.4 v2:172.21.15.150:6808/2997946868 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fcb700716f0 con 0x7fcb7006ec30 2023-12-16T20:48:53.024 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.023+0000 7fcb8e824700 1 -- 172.21.15.52:0/3709669600 --> [v2:172.21.15.150:6808/2997946868,v1:172.21.15.150:6809/2997946868] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcb88072cc0 con 0x7fcb7006ec30 2023-12-16T20:48:53.024 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.023+0000 7fcb84ff9700 1 -- 172.21.15.52:0/3709669600 <== osd.4 v2:172.21.15.150:6808/2997946868 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcb88072cc0 con 0x7fcb7006ec30 2023-12-16T20:48:53.025 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.024+0000 7fcb8e824700 1 -- 172.21.15.52:0/3709669600 >> [v2:172.21.15.150:6808/2997946868,v1:172.21.15.150:6809/2997946868] conn(0x7fcb7006ec30 msgr2=0x7fcb70071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:53.025 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.024+0000 7fcb8e824700 1 --2- 172.21.15.52:0/3709669600 >> [v2:172.21.15.150:6808/2997946868,v1:172.21.15.150:6809/2997946868] conn(0x7fcb7006ec30 0x7fcb70071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.025 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.024+0000 7fcb8e824700 1 -- 172.21.15.52:0/3709669600 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fcb70068f60 msgr2=0x7fcb7006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:53.025 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.024+0000 7fcb8e824700 1 --2- 172.21.15.52:0/3709669600 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fcb70068f60 0x7fcb7006b410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c006fd0 tx=0x7fcb7c006db0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.025 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.024+0000 7fcb8e824700 1 -- 172.21.15.52:0/3709669600 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb880fb1e0 msgr2=0x7fcb880fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:53.025 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.024+0000 7fcb8e824700 1 --2- 172.21.15.52:0/3709669600 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb880fb1e0 0x7fcb880fc3c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009180 tx=0x7fcb78009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:53.025 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.024+0000 7fcb8e824700 1 -- 172.21.15.52:0/3709669600 shutdown_connections 2023-12-16T20:48:53.026 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.024+0000 7fcb8e824700 1 --2- 172.21.15.52:0/3709669600 >> [v2:172.21.15.150:6808/2997946868,v1:172.21.15.150:6809/2997946868] conn(0x7fcb7006ec30 0x7fcb70071040 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.026 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.024+0000 7fcb8e824700 1 --2- 172.21.15.52:0/3709669600 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fcb70068f60 0x7fcb7006b410 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.026 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.024+0000 7fcb8e824700 1 --2- 172.21.15.52:0/3709669600 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb880fb1e0 0x7fcb880fc3c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.026 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.024+0000 7fcb8e824700 1 --2- 172.21.15.52:0/3709669600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb88102c70 0x7fcb880fc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.026 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.024+0000 7fcb8e824700 1 -- 172.21.15.52:0/3709669600 >> 172.21.15.52:0/3709669600 conn(0x7fcb880711a0 msgr2=0x7fcb880fa030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:53.026 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.024+0000 7fcb8e824700 1 -- 172.21.15.52:0/3709669600 shutdown_connections 2023-12-16T20:48:53.026 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.024+0000 7fcb8e824700 1 -- 172.21.15.52:0/3709669600 wait complete. 2023-12-16T20:48:53.241 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:52 smithi052 ceph-mon[80761]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:53.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:52 smithi150 ceph-mon[88000]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:53.555 INFO:teuthology.orchestra.run.smithi052.stdout:137438953483 2023-12-16T20:48:53.556 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.4 2023-12-16T20:48:53.780 INFO:teuthology.orchestra.run.smithi052.stdout:60129542164 2023-12-16T20:48:53.781 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.1 2023-12-16T20:48:53.815 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.463+0000 7ff24b40f700 1 -- 172.21.15.52:0/2786326544 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff244102c70 msgr2=0x7ff244103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:53.815 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.463+0000 7ff24b40f700 1 --2- 172.21.15.52:0/2786326544 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff244102c70 0x7ff244103040 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff234009ce0 tx=0x7ff234004d10 comp rx=0 tx=0).stop 2023-12-16T20:48:53.815 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.463+0000 7ff24b40f700 1 -- 172.21.15.52:0/2786326544 shutdown_connections 2023-12-16T20:48:53.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.463+0000 7ff24b40f700 1 --2- 172.21.15.52:0/2786326544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2440fb1e0 0x7ff2440fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.463+0000 7ff24b40f700 1 --2- 172.21.15.52:0/2786326544 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff244102c70 0x7ff244103040 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.463+0000 7ff24b40f700 1 -- 172.21.15.52:0/2786326544 >> 172.21.15.52:0/2786326544 conn(0x7ff2440711a0 msgr2=0x7ff2440715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:53.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.464+0000 7ff24b40f700 1 -- 172.21.15.52:0/2786326544 shutdown_connections 2023-12-16T20:48:53.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.464+0000 7ff24b40f700 1 -- 172.21.15.52:0/2786326544 wait complete. 2023-12-16T20:48:53.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.464+0000 7ff24b40f700 1 Processor -- start 2023-12-16T20:48:53.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.464+0000 7ff24b40f700 1 -- start start 2023-12-16T20:48:53.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.464+0000 7ff24b40f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2440fb1e0 0x7ff244113770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:53.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.464+0000 7ff24b40f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff244102c70 0x7ff244113cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:53.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.464+0000 7ff24b40f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff244114300 con 0x7ff244102c70 2023-12-16T20:48:53.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.464+0000 7ff24b40f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24410d630 con 0x7ff2440fb1e0 2023-12-16T20:48:53.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.464+0000 7ff2489aa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff244102c70 0x7ff244113cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:53.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.464+0000 7ff2489aa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff244102c70 0x7ff244113cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35428/0 (socket says 172.21.15.52:35428) 2023-12-16T20:48:53.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.464+0000 7ff2489aa700 1 -- 172.21.15.52:0/3619827632 learned_addr learned my addr 172.21.15.52:0/3619827632 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:53.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.464+0000 7ff2491ab700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2440fb1e0 0x7ff244113770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:53.817 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.465+0000 7ff2489aa700 1 -- 172.21.15.52:0/3619827632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2440fb1e0 msgr2=0x7ff244113770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:53.817 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.465+0000 7ff2489aa700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2440fb1e0 0x7ff244113770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.817 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.465+0000 7ff2489aa700 1 -- 172.21.15.52:0/3619827632 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff234009940 con 0x7ff244102c70 2023-12-16T20:48:53.817 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.465+0000 7ff2489aa700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff244102c70 0x7ff244113cb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff238002a50 tx=0x7ff2380046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:53.817 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.465+0000 7ff241ffb700 1 -- 172.21.15.52:0/3619827632 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff238007cc0 con 0x7ff244102c70 2023-12-16T20:48:53.817 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.465+0000 7ff241ffb700 1 -- 172.21.15.52:0/3619827632 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff23800f040 con 0x7ff244102c70 2023-12-16T20:48:53.817 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.465+0000 7ff24b40f700 1 -- 172.21.15.52:0/3619827632 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24410d910 con 0x7ff244102c70 2023-12-16T20:48:53.817 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.465+0000 7ff241ffb700 1 -- 172.21.15.52:0/3619827632 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff23800d9c0 con 0x7ff244102c70 2023-12-16T20:48:53.817 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.465+0000 7ff24b40f700 1 -- 172.21.15.52:0/3619827632 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24410de50 con 0x7ff244102c70 2023-12-16T20:48:53.817 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.466+0000 7ff24b40f700 1 -- 172.21.15.52:0/3619827632 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff24410e370 con 0x7ff244102c70 2023-12-16T20:48:53.817 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.467+0000 7ff241ffb700 1 -- 172.21.15.52:0/3619827632 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7ff23800db20 con 0x7ff244102c70 2023-12-16T20:48:53.818 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.467+0000 7ff241ffb700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7ff22c068f60 0x7ff22c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:53.818 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.467+0000 7ff241ffb700 1 -- 172.21.15.52:0/3619827632 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff238086790 con 0x7ff244102c70 2023-12-16T20:48:53.818 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.467+0000 7ff2491ab700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7ff22c068f60 0x7ff22c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:53.818 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.467+0000 7ff241ffb700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.150:6800/2861125970,v1:172.21.15.150:6801/2861125970] conn(0x7ff22c06ec30 0x7ff22c071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:53.818 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.467+0000 7ff241ffb700 1 -- 172.21.15.52:0/3619827632 --> [v2:172.21.15.150:6800/2861125970,v1:172.21.15.150:6801/2861125970] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff22c0716f0 con 0x7ff22c06ec30 2023-12-16T20:48:53.818 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.467+0000 7ff241ffb700 1 -- 172.21.15.52:0/3619827632 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff23808b220 con 0x7ff244102c70 2023-12-16T20:48:53.818 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.468+0000 7ff2491ab700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7ff22c068f60 0x7ff22c06b410 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff234003400 tx=0x7ff234004d10 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:53.818 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.468+0000 7ff2499ac700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.150:6800/2861125970,v1:172.21.15.150:6801/2861125970] conn(0x7ff22c06ec30 0x7ff22c071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:53.819 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.468+0000 7ff2499ac700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.150:6800/2861125970,v1:172.21.15.150:6801/2861125970] conn(0x7ff22c06ec30 0x7ff22c071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:53.819 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.469+0000 7ff241ffb700 1 -- 172.21.15.52:0/3619827632 <== osd.3 v2:172.21.15.150:6800/2861125970 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7ff22c0716f0 con 0x7ff22c06ec30 2023-12-16T20:48:53.819 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.484+0000 7ff24b40f700 1 -- 172.21.15.52:0/3619827632 --> [v2:172.21.15.150:6800/2861125970,v1:172.21.15.150:6801/2861125970] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff24405e340 con 0x7ff22c06ec30 2023-12-16T20:48:53.819 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.485+0000 7ff241ffb700 1 -- 172.21.15.52:0/3619827632 <== osd.3 v2:172.21.15.150:6800/2861125970 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff24405e340 con 0x7ff22c06ec30 2023-12-16T20:48:53.819 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.485+0000 7ff24b40f700 1 -- 172.21.15.52:0/3619827632 >> [v2:172.21.15.150:6800/2861125970,v1:172.21.15.150:6801/2861125970] conn(0x7ff22c06ec30 msgr2=0x7ff22c071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:53.819 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.485+0000 7ff24b40f700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.150:6800/2861125970,v1:172.21.15.150:6801/2861125970] conn(0x7ff22c06ec30 0x7ff22c071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.819 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.485+0000 7ff24b40f700 1 -- 172.21.15.52:0/3619827632 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7ff22c068f60 msgr2=0x7ff22c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:53.819 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.485+0000 7ff24b40f700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7ff22c068f60 0x7ff22c06b410 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff234003400 tx=0x7ff234004d10 comp rx=0 tx=0).stop 2023-12-16T20:48:53.819 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.485+0000 7ff24b40f700 1 -- 172.21.15.52:0/3619827632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff244102c70 msgr2=0x7ff244113cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:53.819 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.485+0000 7ff24b40f700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff244102c70 0x7ff244113cb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff238002a50 tx=0x7ff2380046b0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.819 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.485+0000 7ff24b40f700 1 -- 172.21.15.52:0/3619827632 shutdown_connections 2023-12-16T20:48:53.819 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.485+0000 7ff24b40f700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.150:6800/2861125970,v1:172.21.15.150:6801/2861125970] conn(0x7ff22c06ec30 0x7ff22c071040 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.819 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.485+0000 7ff24b40f700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7ff22c068f60 0x7ff22c06b410 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.819 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.485+0000 7ff24b40f700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2440fb1e0 0x7ff244113770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.820 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.485+0000 7ff24b40f700 1 --2- 172.21.15.52:0/3619827632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff244102c70 0x7ff244113cb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:53.820 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.485+0000 7ff24b40f700 1 -- 172.21.15.52:0/3619827632 >> 172.21.15.52:0/3619827632 conn(0x7ff2440711a0 msgr2=0x7ff2440fa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:53.820 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.485+0000 7ff24b40f700 1 -- 172.21.15.52:0/3619827632 shutdown_connections 2023-12-16T20:48:53.820 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:53.485+0000 7ff24b40f700 1 -- 172.21.15.52:0/3619827632 wait complete. 2023-12-16T20:48:54.181 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.178+0000 7f57737d4700 1 -- 172.21.15.52:0/3355218965 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f576c0fbb40 msgr2=0x7f576c0fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:54.181 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.178+0000 7f57737d4700 1 --2- 172.21.15.52:0/3355218965 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f576c0fbb40 0x7f576c0fbfb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f575c004940 tx=0x7f575c009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:54.181 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.178+0000 7f57737d4700 1 -- 172.21.15.52:0/3355218965 shutdown_connections 2023-12-16T20:48:54.181 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.178+0000 7f57737d4700 1 --2- 172.21.15.52:0/3355218965 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f576c0fbb40 0x7f576c0fbfb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.181 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.178+0000 7f57737d4700 1 --2- 172.21.15.52:0/3355218965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f576c101b00 0x7f576c101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.181 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.178+0000 7f57737d4700 1 -- 172.21.15.52:0/3355218965 >> 172.21.15.52:0/3355218965 conn(0x7f576c0f76c0 msgr2=0x7f576c0f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:54.181 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.179+0000 7f57737d4700 1 -- 172.21.15.52:0/3355218965 shutdown_connections 2023-12-16T20:48:54.181 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.179+0000 7f57737d4700 1 -- 172.21.15.52:0/3355218965 wait complete. 2023-12-16T20:48:54.181 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.179+0000 7f57737d4700 1 Processor -- start 2023-12-16T20:48:54.182 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.179+0000 7f57737d4700 1 -- start start 2023-12-16T20:48:54.182 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.179+0000 7f57737d4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f576c0fbb40 0x7f576c0fd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:54.182 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.179+0000 7f57737d4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f576c101b00 0x7f576c0fdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:54.182 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.179+0000 7f57737d4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c0fe2f0 con 0x7f576c101b00 2023-12-16T20:48:54.182 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.179+0000 7f57737d4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c0fe460 con 0x7f576c0fbb40 2023-12-16T20:48:54.182 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.179+0000 7f5770d6f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f576c101b00 0x7f576c0fdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:54.182 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.179+0000 7f5770d6f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f576c101b00 0x7f576c0fdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35440/0 (socket says 172.21.15.52:35440) 2023-12-16T20:48:54.182 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.180+0000 7f5770d6f700 1 -- 172.21.15.52:0/1382033349 learned_addr learned my addr 172.21.15.52:0/1382033349 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:54.182 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.180+0000 7f5771570700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f576c0fbb40 0x7f576c0fd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:54.182 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.180+0000 7f5770d6f700 1 -- 172.21.15.52:0/1382033349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f576c0fbb40 msgr2=0x7f576c0fd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:54.182 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.180+0000 7f5770d6f700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f576c0fbb40 0x7f576c0fd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.182 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.180+0000 7f5770d6f700 1 -- 172.21.15.52:0/1382033349 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c005040 con 0x7f576c101b00 2023-12-16T20:48:54.182 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.180+0000 7f5770d6f700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f576c101b00 0x7f576c0fdc60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f575c009180 tx=0x7f575c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:54.182 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.180+0000 7f5761ffb700 1 -- 172.21.15.52:0/1382033349 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f575c00e040 con 0x7f576c101b00 2023-12-16T20:48:54.182 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.180+0000 7f5761ffb700 1 -- 172.21.15.52:0/1382033349 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f575c013420 con 0x7f576c101b00 2023-12-16T20:48:54.183 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.180+0000 7f57737d4700 1 -- 172.21.15.52:0/1382033349 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576c1198c0 con 0x7f576c101b00 2023-12-16T20:48:54.183 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.180+0000 7f5761ffb700 1 -- 172.21.15.52:0/1382033349 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f575c01c630 con 0x7f576c101b00 2023-12-16T20:48:54.183 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.180+0000 7f57737d4700 1 -- 172.21.15.52:0/1382033349 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576c119db0 con 0x7f576c101b00 2023-12-16T20:48:54.184 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.181+0000 7f57737d4700 1 -- 172.21.15.52:0/1382033349 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f576c066710 con 0x7f576c101b00 2023-12-16T20:48:54.184 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.182+0000 7f5761ffb700 1 -- 172.21.15.52:0/1382033349 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f575c00bb80 con 0x7f576c101b00 2023-12-16T20:48:54.184 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.182+0000 7f5761ffb700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5758068f60 0x7f575806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:54.184 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.182+0000 7f5771570700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5758068f60 0x7f575806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:54.184 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.182+0000 7f5761ffb700 1 -- 172.21.15.52:0/1382033349 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f575c087cd0 con 0x7f576c101b00 2023-12-16T20:48:54.184 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.183+0000 7f5761ffb700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.150:6816/4046379778,v1:172.21.15.150:6817/4046379778] conn(0x7f575806ec30 0x7f5758071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:54.184 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.183+0000 7f5761ffb700 1 -- 172.21.15.52:0/1382033349 --> [v2:172.21.15.150:6816/4046379778,v1:172.21.15.150:6817/4046379778] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57580716f0 con 0x7f575806ec30 2023-12-16T20:48:54.184 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.183+0000 7f5761ffb700 1 -- 172.21.15.52:0/1382033349 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f575c08a340 con 0x7f576c101b00 2023-12-16T20:48:54.184 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.183+0000 7f5771570700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5758068f60 0x7f575806b410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5768006fd0 tx=0x7f5768008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:54.184 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.183+0000 7f5771d71700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.150:6816/4046379778,v1:172.21.15.150:6817/4046379778] conn(0x7f575806ec30 0x7f5758071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:54.184 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.183+0000 7f5771d71700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.150:6816/4046379778,v1:172.21.15.150:6817/4046379778] conn(0x7f575806ec30 0x7f5758071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:54.185 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.184+0000 7f5761ffb700 1 -- 172.21.15.52:0/1382033349 <== osd.5 v2:172.21.15.150:6816/4046379778 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f57580716f0 con 0x7f575806ec30 2023-12-16T20:48:54.200 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.199+0000 7f57737d4700 1 -- 172.21.15.52:0/1382033349 --> [v2:172.21.15.150:6816/4046379778,v1:172.21.15.150:6817/4046379778] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f576c0fb2c0 con 0x7f575806ec30 2023-12-16T20:48:54.201 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.200+0000 7f5761ffb700 1 -- 172.21.15.52:0/1382033349 <== osd.5 v2:172.21.15.150:6816/4046379778 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f576c0fb2c0 con 0x7f575806ec30 2023-12-16T20:48:54.201 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.200+0000 7f57737d4700 1 -- 172.21.15.52:0/1382033349 >> [v2:172.21.15.150:6816/4046379778,v1:172.21.15.150:6817/4046379778] conn(0x7f575806ec30 msgr2=0x7f5758071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:54.201 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.200+0000 7f57737d4700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.150:6816/4046379778,v1:172.21.15.150:6817/4046379778] conn(0x7f575806ec30 0x7f5758071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.202 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.200+0000 7f57737d4700 1 -- 172.21.15.52:0/1382033349 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5758068f60 msgr2=0x7f575806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:54.202 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.200+0000 7f57737d4700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5758068f60 0x7f575806b410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5768006fd0 tx=0x7f5768008040 comp rx=0 tx=0).stop 2023-12-16T20:48:54.202 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.200+0000 7f57737d4700 1 -- 172.21.15.52:0/1382033349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f576c101b00 msgr2=0x7f576c0fdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:54.202 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.200+0000 7f57737d4700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f576c101b00 0x7f576c0fdc60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f575c009180 tx=0x7f575c009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:54.202 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.201+0000 7f57737d4700 1 -- 172.21.15.52:0/1382033349 shutdown_connections 2023-12-16T20:48:54.202 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.201+0000 7f57737d4700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.150:6816/4046379778,v1:172.21.15.150:6817/4046379778] conn(0x7f575806ec30 0x7f5758071040 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.202 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.201+0000 7f57737d4700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5758068f60 0x7f575806b410 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.202 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.201+0000 7f57737d4700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f576c0fbb40 0x7f576c0fd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.202 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.201+0000 7f57737d4700 1 --2- 172.21.15.52:0/1382033349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f576c101b00 0x7f576c0fdc60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.202 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.201+0000 7f57737d4700 1 -- 172.21.15.52:0/1382033349 >> 172.21.15.52:0/1382033349 conn(0x7f576c0f76c0 msgr2=0x7f576c065d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:54.202 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.201+0000 7f57737d4700 1 -- 172.21.15.52:0/1382033349 shutdown_connections 2023-12-16T20:48:54.202 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.201+0000 7f57737d4700 1 -- 172.21.15.52:0/1382033349 wait complete. 2023-12-16T20:48:54.302 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.301+0000 7f0795866700 1 -- 172.21.15.52:0/2402374730 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f07900fbb40 msgr2=0x7f07900fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:54.340 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.301+0000 7f0795866700 1 --2- 172.21.15.52:0/2402374730 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f07900fbb40 0x7f07900fbfb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0784004940 tx=0x7f0784009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:54.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.301+0000 7f0795866700 1 -- 172.21.15.52:0/2402374730 shutdown_connections 2023-12-16T20:48:54.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.301+0000 7f0795866700 1 --2- 172.21.15.52:0/2402374730 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f07900fbb40 0x7f07900fbfb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.301+0000 7f0795866700 1 --2- 172.21.15.52:0/2402374730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0790101b00 0x7f0790101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.301+0000 7f0795866700 1 -- 172.21.15.52:0/2402374730 >> 172.21.15.52:0/2402374730 conn(0x7f07900719b0 msgr2=0x7f0790071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:54.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.301+0000 7f0795866700 1 -- 172.21.15.52:0/2402374730 shutdown_connections 2023-12-16T20:48:54.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.301+0000 7f0795866700 1 -- 172.21.15.52:0/2402374730 wait complete. 2023-12-16T20:48:54.344 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.302+0000 7f0795866700 1 Processor -- start 2023-12-16T20:48:54.344 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.302+0000 7f0795866700 1 -- start start 2023-12-16T20:48:54.344 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.302+0000 7f0795866700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07900fbb40 0x7f07900fd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:54.344 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.302+0000 7f0795866700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0790101b00 0x7f07900fdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:54.344 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.302+0000 7f0795866700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07900fe2d0 con 0x7f0790101b00 2023-12-16T20:48:54.344 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.302+0000 7f0795866700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07900fe440 con 0x7f07900fbb40 2023-12-16T20:48:54.344 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.302+0000 7f078e7fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0790101b00 0x7f07900fdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.302+0000 7f078e7fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0790101b00 0x7f07900fdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35458/0 (socket says 172.21.15.52:35458) 2023-12-16T20:48:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.302+0000 7f078e7fc700 1 -- 172.21.15.52:0/1348705609 learned_addr learned my addr 172.21.15.52:0/1348705609 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.302+0000 7f078effd700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07900fbb40 0x7f07900fd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.303+0000 7f078e7fc700 1 -- 172.21.15.52:0/1348705609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07900fbb40 msgr2=0x7f07900fd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.303+0000 7f078e7fc700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07900fbb40 0x7f07900fd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.303+0000 7f078e7fc700 1 -- 172.21.15.52:0/1348705609 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0784005040 con 0x7f0790101b00 2023-12-16T20:48:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.303+0000 7f078e7fc700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0790101b00 0x7f07900fdc80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f078400b3b0 tx=0x7f0784009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.303+0000 7f07777fe700 1 -- 172.21.15.52:0/1348705609 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f078400e040 con 0x7f0790101b00 2023-12-16T20:48:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.303+0000 7f07777fe700 1 -- 172.21.15.52:0/1348705609 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f078401c430 con 0x7f0790101b00 2023-12-16T20:48:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.303+0000 7f0795866700 1 -- 172.21.15.52:0/1348705609 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0790119900 con 0x7f0790101b00 2023-12-16T20:48:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.303+0000 7f0795866700 1 -- 172.21.15.52:0/1348705609 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0790119de0 con 0x7f0790101b00 2023-12-16T20:48:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.303+0000 7f07777fe700 1 -- 172.21.15.52:0/1348705609 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0784013640 con 0x7f0790101b00 2023-12-16T20:48:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.304+0000 7f0795866700 1 -- 172.21.15.52:0/1348705609 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0790100af0 con 0x7f0790101b00 2023-12-16T20:48:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.305+0000 7f07777fe700 1 -- 172.21.15.52:0/1348705609 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f078401c5a0 con 0x7f0790101b00 2023-12-16T20:48:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.305+0000 7f07777fe700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f0778068f60 0x7f077806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.305+0000 7f07777fe700 1 -- 172.21.15.52:0/1348705609 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f07840874f0 con 0x7f0790101b00 2023-12-16T20:48:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.305+0000 7f078effd700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f0778068f60 0x7f077806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.305+0000 7f07777fe700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:6818/4281298930,v1:172.21.15.52:6819/4281298930] conn(0x7f077806ec30 0x7f0778071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.305+0000 7f07777fe700 1 -- 172.21.15.52:0/1348705609 --> [v2:172.21.15.52:6818/4281298930,v1:172.21.15.52:6819/4281298930] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f07780716f0 con 0x7f077806ec30 2023-12-16T20:48:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.305+0000 7f07777fe700 1 -- 172.21.15.52:0/1348705609 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0784089ae0 con 0x7f0790101b00 2023-12-16T20:48:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.306+0000 7f078f7fe700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:6818/4281298930,v1:172.21.15.52:6819/4281298930] conn(0x7f077806ec30 0x7f0778071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.306+0000 7f078effd700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f0778068f60 0x7f077806b410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0780004f60 tx=0x7f078000b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.306+0000 7f078f7fe700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:6818/4281298930,v1:172.21.15.52:6819/4281298930] conn(0x7f077806ec30 0x7f0778071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.307+0000 7f07777fe700 1 -- 172.21.15.52:0/1348705609 <== osd.2 v2:172.21.15.52:6818/4281298930 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f07780716f0 con 0x7f077806ec30 2023-12-16T20:48:54.347 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.322+0000 7f0795866700 1 -- 172.21.15.52:0/1348705609 --> [v2:172.21.15.52:6818/4281298930,v1:172.21.15.52:6819/4281298930] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0790100ed0 con 0x7f077806ec30 2023-12-16T20:48:54.347 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.322+0000 7f07777fe700 1 -- 172.21.15.52:0/1348705609 <== osd.2 v2:172.21.15.52:6818/4281298930 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0790100ed0 con 0x7f077806ec30 2023-12-16T20:48:54.348 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.322+0000 7f0795866700 1 -- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:6818/4281298930,v1:172.21.15.52:6819/4281298930] conn(0x7f077806ec30 msgr2=0x7f0778071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:54.348 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.322+0000 7f0795866700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:6818/4281298930,v1:172.21.15.52:6819/4281298930] conn(0x7f077806ec30 0x7f0778071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.348 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.322+0000 7f0795866700 1 -- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f0778068f60 msgr2=0x7f077806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:54.348 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.322+0000 7f0795866700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f0778068f60 0x7f077806b410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0780004f60 tx=0x7f078000b040 comp rx=0 tx=0).stop 2023-12-16T20:48:54.348 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.322+0000 7f0795866700 1 -- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0790101b00 msgr2=0x7f07900fdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:54.348 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.322+0000 7f0795866700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0790101b00 0x7f07900fdc80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f078400b3b0 tx=0x7f0784009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:54.348 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.322+0000 7f0795866700 1 -- 172.21.15.52:0/1348705609 shutdown_connections 2023-12-16T20:48:54.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.322+0000 7f0795866700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:6818/4281298930,v1:172.21.15.52:6819/4281298930] conn(0x7f077806ec30 0x7f0778071040 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.322+0000 7f0795866700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f0778068f60 0x7f077806b410 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.322+0000 7f0795866700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07900fbb40 0x7f07900fd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.322+0000 7f0795866700 1 --2- 172.21.15.52:0/1348705609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0790101b00 0x7f07900fdc80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:54.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.322+0000 7f0795866700 1 -- 172.21.15.52:0/1348705609 >> 172.21.15.52:0/1348705609 conn(0x7f07900719b0 msgr2=0x7f07900654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:54.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.323+0000 7f0795866700 1 -- 172.21.15.52:0/1348705609 shutdown_connections 2023-12-16T20:48:54.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:54.323+0000 7f0795866700 1 -- 172.21.15.52:0/1348705609 wait complete. 2023-12-16T20:48:54.431 INFO:teuthology.orchestra.run.smithi052.stdout:107374182413 2023-12-16T20:48:54.432 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.3 2023-12-16T20:48:54.800 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:55.232 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:54 smithi052 ceph-mon[80761]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:55.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:54 smithi150 ceph-mon[88000]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:55.490 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:55.557 INFO:teuthology.orchestra.run.smithi052.stdout:154618822664 2023-12-16T20:48:55.558 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.5 2023-12-16T20:48:55.754 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:55.773 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:56.194 INFO:teuthology.orchestra.run.smithi052.stdout:81604378641 2023-12-16T20:48:56.195 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.2 2023-12-16T20:48:57.032 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:56 smithi052 ceph-mon[80761]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:57.094 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.093+0000 7fc1dec91700 1 -- 172.21.15.52:0/4121037483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d8102c00 msgr2=0x7fc1d8102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:57.094 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.093+0000 7fc1dec91700 1 --2- 172.21.15.52:0/4121037483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d8102c00 0x7fc1d8102fd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8004940 tx=0x7fc1c8009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:57.094 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.093+0000 7fc1dec91700 1 -- 172.21.15.52:0/4121037483 shutdown_connections 2023-12-16T20:48:57.095 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.093+0000 7fc1dec91700 1 --2- 172.21.15.52:0/4121037483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d81035a0 0x7fc1d80fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:57.095 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.093+0000 7fc1dec91700 1 --2- 172.21.15.52:0/4121037483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d8102c00 0x7fc1d8102fd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:57.095 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.093+0000 7fc1dec91700 1 -- 172.21.15.52:0/4121037483 >> 172.21.15.52:0/4121037483 conn(0x7fc1d80f8790 msgr2=0x7fc1d80faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:57.095 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.093+0000 7fc1dec91700 1 -- 172.21.15.52:0/4121037483 shutdown_connections 2023-12-16T20:48:57.095 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.093+0000 7fc1dec91700 1 -- 172.21.15.52:0/4121037483 wait complete. 2023-12-16T20:48:57.095 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.094+0000 7fc1dec91700 1 Processor -- start 2023-12-16T20:48:57.095 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.094+0000 7fc1dec91700 1 -- start start 2023-12-16T20:48:57.096 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.094+0000 7fc1dec91700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d8102c00 0x7fc1d8113610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:57.096 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.094+0000 7fc1dec91700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d81035a0 0x7fc1d8113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:57.096 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.094+0000 7fc1dec91700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d81141e0 con 0x7fc1d81035a0 2023-12-16T20:48:57.096 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.094+0000 7fc1dec91700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d810d4d0 con 0x7fc1d8102c00 2023-12-16T20:48:57.096 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.094+0000 7fc1d7fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d81035a0 0x7fc1d8113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.094+0000 7fc1d7fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d81035a0 0x7fc1d8113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35476/0 (socket says 172.21.15.52:35476) 2023-12-16T20:48:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.094+0000 7fc1d7fff700 1 -- 172.21.15.52:0/65151679 learned_addr learned my addr 172.21.15.52:0/65151679 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.094+0000 7fc1dca2d700 1 --2- 172.21.15.52:0/65151679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d8102c00 0x7fc1d8113610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.095+0000 7fc1d7fff700 1 -- 172.21.15.52:0/65151679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d8102c00 msgr2=0x7fc1d8113610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.095+0000 7fc1d7fff700 1 --2- 172.21.15.52:0/65151679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d8102c00 0x7fc1d8113610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.095+0000 7fc1d7fff700 1 -- 172.21.15.52:0/65151679 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c8005040 con 0x7fc1d81035a0 2023-12-16T20:48:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.095+0000 7fc1d7fff700 1 --2- 172.21.15.52:0/65151679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d81035a0 0x7fc1d8113b90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc002a50 tx=0x7fc1cc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.095+0000 7fc1d57fa700 1 -- 172.21.15.52:0/65151679 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1cc007cc0 con 0x7fc1d81035a0 2023-12-16T20:48:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.095+0000 7fc1d57fa700 1 -- 172.21.15.52:0/65151679 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc1cc00f040 con 0x7fc1d81035a0 2023-12-16T20:48:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.095+0000 7fc1dec91700 1 -- 172.21.15.52:0/65151679 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d810d760 con 0x7fc1d81035a0 2023-12-16T20:48:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.095+0000 7fc1d57fa700 1 -- 172.21.15.52:0/65151679 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1cc00d920 con 0x7fc1d81035a0 2023-12-16T20:48:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.095+0000 7fc1dec91700 1 -- 172.21.15.52:0/65151679 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d810dca0 con 0x7fc1d81035a0 2023-12-16T20:48:57.098 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.096+0000 7fc1dec91700 1 -- 172.21.15.52:0/65151679 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d80fff10 con 0x7fc1d81035a0 2023-12-16T20:48:57.098 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.096+0000 7fc1d57fa700 1 -- 172.21.15.52:0/65151679 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fc1cc009680 con 0x7fc1d81035a0 2023-12-16T20:48:57.098 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.097+0000 7fc1d57fa700 1 --2- 172.21.15.52:0/65151679 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc1c0068f60 0x7fc1c006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:57.098 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.097+0000 7fc1d57fa700 1 -- 172.21.15.52:0/65151679 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc1cc0866f0 con 0x7fc1d81035a0 2023-12-16T20:48:57.098 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.097+0000 7fc1dca2d700 1 --2- 172.21.15.52:0/65151679 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc1c0068f60 0x7fc1c006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:57.098 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.097+0000 7fc1dca2d700 1 --2- 172.21.15.52:0/65151679 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc1c0068f60 0x7fc1c006b410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc1c800b3b0 tx=0x7fc1c8009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:57.102 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.101+0000 7fc1d57fa700 1 -- 172.21.15.52:0/65151679 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc1cc056990 con 0x7fc1d81035a0 2023-12-16T20:48:57.302 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.301+0000 7fc1dec91700 1 -- 172.21.15.52:0/65151679 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc1d80fc390 con 0x7fc1d81035a0 2023-12-16T20:48:57.302 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.302+0000 7fc1d57fa700 1 -- 172.21.15.52:0/65151679 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc1cc056520 con 0x7fc1d81035a0 2023-12-16T20:48:57.303 INFO:teuthology.orchestra.run.smithi052.stdout:38654705688 2023-12-16T20:48:57.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.304+0000 7fc1dec91700 1 -- 172.21.15.52:0/65151679 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc1c0068f60 msgr2=0x7fc1c006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:57.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.304+0000 7fc1dec91700 1 --2- 172.21.15.52:0/65151679 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc1c0068f60 0x7fc1c006b410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc1c800b3b0 tx=0x7fc1c8009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:57.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.304+0000 7fc1dec91700 1 -- 172.21.15.52:0/65151679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d81035a0 msgr2=0x7fc1d8113b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:57.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.304+0000 7fc1dec91700 1 --2- 172.21.15.52:0/65151679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d81035a0 0x7fc1d8113b90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc002a50 tx=0x7fc1cc004600 comp rx=0 tx=0).stop 2023-12-16T20:48:57.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.304+0000 7fc1dec91700 1 -- 172.21.15.52:0/65151679 shutdown_connections 2023-12-16T20:48:57.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.304+0000 7fc1dec91700 1 --2- 172.21.15.52:0/65151679 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc1c0068f60 0x7fc1c006b410 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:57.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.304+0000 7fc1dec91700 1 --2- 172.21.15.52:0/65151679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d8102c00 0x7fc1d8113610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:57.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.304+0000 7fc1dec91700 1 --2- 172.21.15.52:0/65151679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d81035a0 0x7fc1d8113b90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:57.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.304+0000 7fc1dec91700 1 -- 172.21.15.52:0/65151679 >> 172.21.15.52:0/65151679 conn(0x7fc1d80f8790 msgr2=0x7fc1d80fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:57.306 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.304+0000 7fc1dec91700 1 -- 172.21.15.52:0/65151679 shutdown_connections 2023-12-16T20:48:57.306 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.304+0000 7fc1dec91700 1 -- 172.21.15.52:0/65151679 wait complete. 2023-12-16T20:48:57.309 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.308+0000 7f3ae36c8700 1 -- 172.21.15.52:0/1452355149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3adc102410 msgr2=0x7f3adc0fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:57.309 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.308+0000 7f3ae36c8700 1 --2- 172.21.15.52:0/1452355149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3adc102410 0x7f3adc0fc2f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3acc004940 tx=0x7f3acc009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:57.309 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.308+0000 7f3ae36c8700 1 -- 172.21.15.52:0/1452355149 shutdown_connections 2023-12-16T20:48:57.309 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.308+0000 7f3ae36c8700 1 --2- 172.21.15.52:0/1452355149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3adc102410 0x7f3adc0fc2f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:57.309 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.308+0000 7f3ae36c8700 1 --2- 172.21.15.52:0/1452355149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3adc101b00 0x7f3adc101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:57.309 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.308+0000 7f3ae36c8700 1 -- 172.21.15.52:0/1452355149 >> 172.21.15.52:0/1452355149 conn(0x7f3adc0711a0 msgr2=0x7f3adc0715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:57.309 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.309+0000 7f3ae36c8700 1 -- 172.21.15.52:0/1452355149 shutdown_connections 2023-12-16T20:48:57.310 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.309+0000 7f3ae36c8700 1 -- 172.21.15.52:0/1452355149 wait complete. 2023-12-16T20:48:57.310 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.309+0000 7f3ae36c8700 1 Processor -- start 2023-12-16T20:48:57.310 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.309+0000 7f3ae36c8700 1 -- start start 2023-12-16T20:48:57.311 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.310+0000 7f3ae36c8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3adc101b00 0x7f3adc113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:57.311 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.310+0000 7f3ae36c8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3adc102410 0x7f3adc113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:57.311 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.310+0000 7f3ae36c8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc1141d0 con 0x7f3adc102410 2023-12-16T20:48:57.311 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.310+0000 7f3ae36c8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc10d4c0 con 0x7f3adc101b00 2023-12-16T20:48:57.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.310+0000 7f3ae0c63700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3adc102410 0x7f3adc113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:57.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.310+0000 7f3ae1464700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3adc101b00 0x7f3adc113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:57.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.310+0000 7f3ae1464700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3adc101b00 0x7f3adc113600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.52:55948/0 (socket says 172.21.15.52:55948) 2023-12-16T20:48:57.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.310+0000 7f3ae0c63700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3adc102410 0x7f3adc113b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35486/0 (socket says 172.21.15.52:35486) 2023-12-16T20:48:57.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.310+0000 7f3ae1464700 1 -- 172.21.15.52:0/3224044240 learned_addr learned my addr 172.21.15.52:0/3224044240 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:57.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.310+0000 7f3ae0c63700 1 -- 172.21.15.52:0/3224044240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3adc101b00 msgr2=0x7f3adc113600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:57.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.310+0000 7f3ae0c63700 1 --2- 172.21.15.52:0/3224044240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3adc101b00 0x7f3adc113600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:57.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.310+0000 7f3ae0c63700 1 -- 172.21.15.52:0/3224044240 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc005040 con 0x7f3adc102410 2023-12-16T20:48:57.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.310+0000 7f3ae1464700 1 --2- 172.21.15.52:0/3224044240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3adc101b00 0x7f3adc113600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T20:48:57.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.310+0000 7f3ae0c63700 1 --2- 172.21.15.52:0/3224044240 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3adc102410 0x7f3adc113b80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3acc000c00 tx=0x7f3acc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:57.313 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.311+0000 7f3ad1ffb700 1 -- 172.21.15.52:0/3224044240 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3acc021070 con 0x7f3adc102410 2023-12-16T20:48:57.313 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.311+0000 7f3ae36c8700 1 -- 172.21.15.52:0/3224044240 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3adc10d6f0 con 0x7f3adc102410 2023-12-16T20:48:57.313 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.311+0000 7f3ad1ffb700 1 -- 172.21.15.52:0/3224044240 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3acc01c400 con 0x7f3adc102410 2023-12-16T20:48:57.313 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.311+0000 7f3ad1ffb700 1 -- 172.21.15.52:0/3224044240 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3acc00e070 con 0x7f3adc102410 2023-12-16T20:48:57.313 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.311+0000 7f3ae36c8700 1 -- 172.21.15.52:0/3224044240 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3adc10dbd0 con 0x7f3adc102410 2023-12-16T20:48:57.313 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.313+0000 7f3ae36c8700 1 -- 172.21.15.52:0/3224044240 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3adc0ffe70 con 0x7f3adc102410 2023-12-16T20:48:57.314 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.313+0000 7f3ad1ffb700 1 -- 172.21.15.52:0/3224044240 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f3acc013a30 con 0x7f3adc102410 2023-12-16T20:48:57.314 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.314+0000 7f3ad1ffb700 1 --2- 172.21.15.52:0/3224044240 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f3ac8068f70 0x7f3ac806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:57.315 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.314+0000 7f3ad1ffb700 1 -- 172.21.15.52:0/3224044240 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3acc087b50 con 0x7f3adc102410 2023-12-16T20:48:57.315 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.314+0000 7f3ae1464700 1 --2- 172.21.15.52:0/3224044240 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f3ac8068f70 0x7f3ac806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:57.315 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.315+0000 7f3ae1464700 1 --2- 172.21.15.52:0/3224044240 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f3ac8068f70 0x7f3ac806b420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3adc0f9a20 tx=0x7f3ad800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:57.319 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.319+0000 7f3ad1ffb700 1 -- 172.21.15.52:0/3224044240 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3acc057eb0 con 0x7f3adc102410 2023-12-16T20:48:57.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:56 smithi150 ceph-mon[88000]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:57.535 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.534+0000 7f3ae36c8700 1 -- 172.21.15.52:0/3224044240 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f3adc072cc0 con 0x7f3adc102410 2023-12-16T20:48:57.535 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.534+0000 7f3ad1ffb700 1 -- 172.21.15.52:0/3224044240 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3acc057a40 con 0x7f3adc102410 2023-12-16T20:48:57.535 INFO:teuthology.orchestra.run.smithi052.stdout:137438953483 2023-12-16T20:48:57.537 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.536+0000 7f3ae36c8700 1 -- 172.21.15.52:0/3224044240 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f3ac8068f70 msgr2=0x7f3ac806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:57.537 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.536+0000 7f3ae36c8700 1 --2- 172.21.15.52:0/3224044240 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f3ac8068f70 0x7f3ac806b420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3adc0f9a20 tx=0x7f3ad800b040 comp rx=0 tx=0).stop 2023-12-16T20:48:57.537 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.536+0000 7f3ae36c8700 1 -- 172.21.15.52:0/3224044240 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3adc102410 msgr2=0x7f3adc113b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:57.537 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.536+0000 7f3ae36c8700 1 --2- 172.21.15.52:0/3224044240 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3adc102410 0x7f3adc113b80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3acc000c00 tx=0x7f3acc009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:57.537 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.536+0000 7f3ae36c8700 1 -- 172.21.15.52:0/3224044240 shutdown_connections 2023-12-16T20:48:57.538 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.536+0000 7f3ae36c8700 1 --2- 172.21.15.52:0/3224044240 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f3ac8068f70 0x7f3ac806b420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:57.538 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.536+0000 7f3ae36c8700 1 --2- 172.21.15.52:0/3224044240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3adc101b00 0x7f3adc113600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:57.538 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.536+0000 7f3ae36c8700 1 --2- 172.21.15.52:0/3224044240 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3adc102410 0x7f3adc113b80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:57.538 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.536+0000 7f3ae36c8700 1 -- 172.21.15.52:0/3224044240 >> 172.21.15.52:0/3224044240 conn(0x7f3adc0711a0 msgr2=0x7f3adc106070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:57.538 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.537+0000 7f3ae36c8700 1 -- 172.21.15.52:0/3224044240 shutdown_connections 2023-12-16T20:48:57.538 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.537+0000 7f3ae36c8700 1 -- 172.21.15.52:0/3224044240 wait complete. 2023-12-16T20:48:57.739 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:57.909 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.908+0000 7f03f97d7700 1 -- 172.21.15.52:0/1466241634 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03f40fb210 msgr2=0x7f03f40fb680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:57.909 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.908+0000 7f03f97d7700 1 --2- 172.21.15.52:0/1466241634 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03f40fb210 0x7f03f40fb680 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f03e8004940 tx=0x7f03e8009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:57.909 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.908+0000 7f03f97d7700 1 -- 172.21.15.52:0/1466241634 shutdown_connections 2023-12-16T20:48:57.909 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.908+0000 7f03f97d7700 1 --2- 172.21.15.52:0/1466241634 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03f40fb210 0x7f03f40fb680 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:57.909 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.908+0000 7f03f97d7700 1 --2- 172.21.15.52:0/1466241634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f4102c90 0x7f03f4103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:57.910 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.908+0000 7f03f97d7700 1 -- 172.21.15.52:0/1466241634 >> 172.21.15.52:0/1466241634 conn(0x7f03f40711b0 msgr2=0x7f03f40715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:57.910 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.908+0000 7f03f97d7700 1 -- 172.21.15.52:0/1466241634 shutdown_connections 2023-12-16T20:48:57.910 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.908+0000 7f03f97d7700 1 -- 172.21.15.52:0/1466241634 wait complete. 2023-12-16T20:48:57.911 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.909+0000 7f03f97d7700 1 Processor -- start 2023-12-16T20:48:57.911 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.909+0000 7f03f97d7700 1 -- start start 2023-12-16T20:48:57.911 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.909+0000 7f03f97d7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03f40fb210 0x7f03f41136a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:57.911 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.909+0000 7f03f97d7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f4102c90 0x7f03f4113be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:57.911 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.909+0000 7f03f97d7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f41142c0 con 0x7f03f40fb210 2023-12-16T20:48:57.911 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.909+0000 7f03f97d7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f410d580 con 0x7f03f4102c90 2023-12-16T20:48:57.911 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.909+0000 7f03f2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03f40fb210 0x7f03f41136a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:57.912 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.909+0000 7f03f2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03f40fb210 0x7f03f41136a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35510/0 (socket says 172.21.15.52:35510) 2023-12-16T20:48:57.912 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.909+0000 7f03f2ffd700 1 -- 172.21.15.52:0/3641847676 learned_addr learned my addr 172.21.15.52:0/3641847676 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:48:57.912 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.909+0000 7f03f27fc700 1 --2- 172.21.15.52:0/3641847676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f4102c90 0x7f03f4113be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:57.912 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.910+0000 7f03f2ffd700 1 -- 172.21.15.52:0/3641847676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f4102c90 msgr2=0x7f03f4113be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:57.912 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.910+0000 7f03f2ffd700 1 --2- 172.21.15.52:0/3641847676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f4102c90 0x7f03f4113be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:57.912 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.910+0000 7f03f2ffd700 1 -- 172.21.15.52:0/3641847676 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e8005040 con 0x7f03f40fb210 2023-12-16T20:48:57.912 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.910+0000 7f03f2ffd700 1 --2- 172.21.15.52:0/3641847676 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03f40fb210 0x7f03f41136a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f03e400b790 tx=0x7f03e4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:57.912 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.910+0000 7f03db7fe700 1 -- 172.21.15.52:0/3641847676 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03e400f6f0 con 0x7f03f40fb210 2023-12-16T20:48:57.912 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.910+0000 7f03db7fe700 1 -- 172.21.15.52:0/3641847676 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03e400fd00 con 0x7f03f40fb210 2023-12-16T20:48:57.912 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.910+0000 7f03f97d7700 1 -- 172.21.15.52:0/3641847676 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f410d860 con 0x7f03f40fb210 2023-12-16T20:48:57.912 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.910+0000 7f03db7fe700 1 -- 172.21.15.52:0/3641847676 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03e4011e90 con 0x7f03f40fb210 2023-12-16T20:48:57.912 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.910+0000 7f03f97d7700 1 -- 172.21.15.52:0/3641847676 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f410dda0 con 0x7f03f40fb210 2023-12-16T20:48:57.913 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.911+0000 7f03f97d7700 1 -- 172.21.15.52:0/3641847676 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f40fff40 con 0x7f03f40fb210 2023-12-16T20:48:57.913 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.911+0000 7f03db7fe700 1 -- 172.21.15.52:0/3641847676 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f03e400f850 con 0x7f03f40fb210 2023-12-16T20:48:57.913 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.912+0000 7f03db7fe700 1 --2- 172.21.15.52:0/3641847676 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f03dc068f60 0x7f03dc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:48:57.913 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.912+0000 7f03db7fe700 1 -- 172.21.15.52:0/3641847676 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f03e4086820 con 0x7f03f40fb210 2023-12-16T20:48:57.913 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.912+0000 7f03f27fc700 1 --2- 172.21.15.52:0/3641847676 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f03dc068f60 0x7f03dc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:48:57.913 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.912+0000 7f03f27fc700 1 --2- 172.21.15.52:0/3641847676 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f03dc068f60 0x7f03dc06b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f03f410ef80 tx=0x7f03e8009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:48:57.917 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:57.916+0000 7f03db7fe700 1 -- 172.21.15.52:0/3641847676 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f03e4056b70 con 0x7f03f40fb210 2023-12-16T20:48:58.123 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:58.122+0000 7f03f97d7700 1 -- 172.21.15.52:0/3641847676 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f03f404bb80 con 0x7f03f40fb210 2023-12-16T20:48:58.123 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:58.122+0000 7f03db7fe700 1 -- 172.21.15.52:0/3641847676 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f03e4056700 con 0x7f03f40fb210 2023-12-16T20:48:58.123 INFO:teuthology.orchestra.run.smithi052.stdout:60129542165 2023-12-16T20:48:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:58.125+0000 7f03f97d7700 1 -- 172.21.15.52:0/3641847676 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f03dc068f60 msgr2=0x7f03dc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:58.125+0000 7f03f97d7700 1 --2- 172.21.15.52:0/3641847676 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f03dc068f60 0x7f03dc06b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f03f410ef80 tx=0x7f03e8009e30 comp rx=0 tx=0).stop 2023-12-16T20:48:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:58.125+0000 7f03f97d7700 1 -- 172.21.15.52:0/3641847676 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03f40fb210 msgr2=0x7f03f41136a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:48:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:58.125+0000 7f03f97d7700 1 --2- 172.21.15.52:0/3641847676 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03f40fb210 0x7f03f41136a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f03e400b790 tx=0x7f03e4009220 comp rx=0 tx=0).stop 2023-12-16T20:48:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:58.125+0000 7f03f97d7700 1 -- 172.21.15.52:0/3641847676 shutdown_connections 2023-12-16T20:48:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:58.125+0000 7f03f97d7700 1 --2- 172.21.15.52:0/3641847676 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f03dc068f60 0x7f03dc06b410 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:58.125+0000 7f03f97d7700 1 --2- 172.21.15.52:0/3641847676 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03f40fb210 0x7f03f41136a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:58.125+0000 7f03f97d7700 1 --2- 172.21.15.52:0/3641847676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f4102c90 0x7f03f4113be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:48:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:58.125+0000 7f03f97d7700 1 -- 172.21.15.52:0/3641847676 >> 172.21.15.52:0/3641847676 conn(0x7f03f40711b0 msgr2=0x7f03f40fa330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:48:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:58.125+0000 7f03f97d7700 1 -- 172.21.15.52:0/3641847676 shutdown_connections 2023-12-16T20:48:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:48:58.125+0000 7f03f97d7700 1 -- 172.21.15.52:0/3641847676 wait complete. 2023-12-16T20:48:58.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:57 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/65151679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-12-16T20:48:58.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:57 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3224044240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-12-16T20:48:58.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:57 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/65151679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-12-16T20:48:58.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:57 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/3224044240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-12-16T20:48:58.695 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:48:58.713 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705687 got 38654705688 for osd.0 2023-12-16T20:48:58.713 DEBUG:teuthology.parallel:result is None 2023-12-16T20:48:58.948 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542164 got 60129542165 for osd.1 2023-12-16T20:48:58.949 DEBUG:teuthology.parallel:result is None 2023-12-16T20:48:59.219 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:58 smithi052 ceph-mon[80761]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:59.219 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:48:58 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3641847676' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-12-16T20:48:59.254 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953483 got 137438953483 for osd.4 2023-12-16T20:48:59.255 DEBUG:teuthology.parallel:result is None 2023-12-16T20:48:59.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:58 smithi150 ceph-mon[88000]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:48:59.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:48:58 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/3641847676' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-12-16T20:49:00.053 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.052+0000 7fa71d0f7700 1 -- 172.21.15.52:0/2785023772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa7180fe570 msgr2=0x7fa718106c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:00.053 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.052+0000 7fa71d0f7700 1 --2- 172.21.15.52:0/2785023772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa7180fe570 0x7fa718106c40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa70c004940 tx=0x7fa70c009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:00.053 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.052+0000 7fa71d0f7700 1 -- 172.21.15.52:0/2785023772 shutdown_connections 2023-12-16T20:49:00.053 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.052+0000 7fa71d0f7700 1 --2- 172.21.15.52:0/2785023772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa7180fe570 0x7fa718106c40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.053 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.052+0000 7fa71d0f7700 1 --2- 172.21.15.52:0/2785023772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7180fdbd0 0x7fa7180fdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.054 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.052+0000 7fa71d0f7700 1 -- 172.21.15.52:0/2785023772 >> 172.21.15.52:0/2785023772 conn(0x7fa7180f7670 msgr2=0x7fa7180f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:00.054 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.052+0000 7fa71d0f7700 1 -- 172.21.15.52:0/2785023772 shutdown_connections 2023-12-16T20:49:00.054 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.052+0000 7fa71d0f7700 1 -- 172.21.15.52:0/2785023772 wait complete. 2023-12-16T20:49:00.055 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.053+0000 7fa71d0f7700 1 Processor -- start 2023-12-16T20:49:00.055 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.053+0000 7fa71d0f7700 1 -- start start 2023-12-16T20:49:00.055 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.053+0000 7fa71d0f7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa7180fdbd0 0x7fa7181158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:00.055 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.053+0000 7fa71d0f7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7180fe570 0x7fa718115e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:00.055 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.053+0000 7fa71d0f7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7181164f0 con 0x7fa7180fdbd0 2023-12-16T20:49:00.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.053+0000 7fa71d0f7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71810f790 con 0x7fa7180fe570 2023-12-16T20:49:00.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.054+0000 7fa716d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa7180fdbd0 0x7fa7181158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:00.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.054+0000 7fa716d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa7180fdbd0 0x7fa7181158d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35528/0 (socket says 172.21.15.52:35528) 2023-12-16T20:49:00.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.054+0000 7fa716d9d700 1 -- 172.21.15.52:0/3163676355 learned_addr learned my addr 172.21.15.52:0/3163676355 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:00.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.054+0000 7fa71659c700 1 --2- 172.21.15.52:0/3163676355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7180fe570 0x7fa718115e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:00.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.054+0000 7fa716d9d700 1 -- 172.21.15.52:0/3163676355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7180fe570 msgr2=0x7fa718115e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:00.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.054+0000 7fa716d9d700 1 --2- 172.21.15.52:0/3163676355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7180fe570 0x7fa718115e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.054+0000 7fa716d9d700 1 -- 172.21.15.52:0/3163676355 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa70c005040 con 0x7fa7180fdbd0 2023-12-16T20:49:00.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.054+0000 7fa716d9d700 1 --2- 172.21.15.52:0/3163676355 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa7180fdbd0 0x7fa7181158d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa70800b790 tx=0x7fa708009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:00.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.054+0000 7fa6ff7fe700 1 -- 172.21.15.52:0/3163676355 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa70800f6f0 con 0x7fa7180fdbd0 2023-12-16T20:49:00.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.054+0000 7fa71d0f7700 1 -- 172.21.15.52:0/3163676355 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa71810fa70 con 0x7fa7180fdbd0 2023-12-16T20:49:00.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.054+0000 7fa6ff7fe700 1 -- 172.21.15.52:0/3163676355 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa70800fd00 con 0x7fa7180fdbd0 2023-12-16T20:49:00.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.054+0000 7fa6ff7fe700 1 -- 172.21.15.52:0/3163676355 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7080103a0 con 0x7fa7180fdbd0 2023-12-16T20:49:00.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.054+0000 7fa71d0f7700 1 -- 172.21.15.52:0/3163676355 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa71810ffb0 con 0x7fa7180fdbd0 2023-12-16T20:49:00.057 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.055+0000 7fa71d0f7700 1 -- 172.21.15.52:0/3163676355 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7181066a0 con 0x7fa7180fdbd0 2023-12-16T20:49:00.057 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.056+0000 7fa6ff7fe700 1 -- 172.21.15.52:0/3163676355 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa70800f850 con 0x7fa7180fdbd0 2023-12-16T20:49:00.057 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.056+0000 7fa6ff7fe700 1 --2- 172.21.15.52:0/3163676355 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fa700068f70 0x7fa70006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:00.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.056+0000 7fa6ff7fe700 1 -- 172.21.15.52:0/3163676355 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa708086830 con 0x7fa7180fdbd0 2023-12-16T20:49:00.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.056+0000 7fa71659c700 1 --2- 172.21.15.52:0/3163676355 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fa700068f70 0x7fa70006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:00.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.057+0000 7fa71659c700 1 --2- 172.21.15.52:0/3163676355 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fa700068f70 0x7fa70006b420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa718111100 tx=0x7fa70c009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:00.061 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.061+0000 7fa6ff7fe700 1 -- 172.21.15.52:0/3163676355 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa708056ad0 con 0x7fa7180fdbd0 2023-12-16T20:49:00.264 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.264+0000 7fa71d0f7700 1 -- 172.21.15.52:0/3163676355 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fa7180fb270 con 0x7fa7180fdbd0 2023-12-16T20:49:00.265 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.264+0000 7fa6ff7fe700 1 -- 172.21.15.52:0/3163676355 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa708056660 con 0x7fa7180fdbd0 2023-12-16T20:49:00.265 INFO:teuthology.orchestra.run.smithi052.stdout:154618822665 2023-12-16T20:49:00.267 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.266+0000 7fa71d0f7700 1 -- 172.21.15.52:0/3163676355 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fa700068f70 msgr2=0x7fa70006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:00.267 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.266+0000 7fa71d0f7700 1 --2- 172.21.15.52:0/3163676355 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fa700068f70 0x7fa70006b420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa718111100 tx=0x7fa70c009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:00.267 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.266+0000 7fa71d0f7700 1 -- 172.21.15.52:0/3163676355 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa7180fdbd0 msgr2=0x7fa7181158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:00.267 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.266+0000 7fa71d0f7700 1 --2- 172.21.15.52:0/3163676355 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa7180fdbd0 0x7fa7181158d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa70800b790 tx=0x7fa708009250 comp rx=0 tx=0).stop 2023-12-16T20:49:00.267 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.266+0000 7fa71d0f7700 1 -- 172.21.15.52:0/3163676355 shutdown_connections 2023-12-16T20:49:00.267 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.266+0000 7fa71d0f7700 1 --2- 172.21.15.52:0/3163676355 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fa700068f70 0x7fa70006b420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.267 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.266+0000 7fa71d0f7700 1 --2- 172.21.15.52:0/3163676355 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa7180fdbd0 0x7fa7181158d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.268 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.266+0000 7fa71d0f7700 1 --2- 172.21.15.52:0/3163676355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7180fe570 0x7fa718115e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.268 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.266+0000 7fa71d0f7700 1 -- 172.21.15.52:0/3163676355 >> 172.21.15.52:0/3163676355 conn(0x7fa7180f7670 msgr2=0x7fa718103820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:00.268 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.266+0000 7fa71d0f7700 1 -- 172.21.15.52:0/3163676355 shutdown_connections 2023-12-16T20:49:00.268 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.266+0000 7fa71d0f7700 1 -- 172.21.15.52:0/3163676355 wait complete. 2023-12-16T20:49:00.405 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.404+0000 7f38ca4f3700 1 -- 172.21.15.52:0/192063555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f38c4102c70 msgr2=0x7f38c4103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:00.405 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.404+0000 7f38ca4f3700 1 --2- 172.21.15.52:0/192063555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f38c4102c70 0x7f38c4103040 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f38b4004940 tx=0x7f38b4009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:00.405 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.404+0000 7f38ca4f3700 1 -- 172.21.15.52:0/192063555 shutdown_connections 2023-12-16T20:49:00.405 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.404+0000 7f38ca4f3700 1 --2- 172.21.15.52:0/192063555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38c40fb1e0 0x7f38c40fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.405 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.404+0000 7f38ca4f3700 1 --2- 172.21.15.52:0/192063555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f38c4102c70 0x7f38c4103040 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.405 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.404+0000 7f38ca4f3700 1 -- 172.21.15.52:0/192063555 >> 172.21.15.52:0/192063555 conn(0x7f38c40711a0 msgr2=0x7f38c40715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:00.405 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.404+0000 7f38ca4f3700 1 -- 172.21.15.52:0/192063555 shutdown_connections 2023-12-16T20:49:00.405 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.404+0000 7f38ca4f3700 1 -- 172.21.15.52:0/192063555 wait complete. 2023-12-16T20:49:00.407 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.405+0000 7f38ca4f3700 1 Processor -- start 2023-12-16T20:49:00.407 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.405+0000 7f38ca4f3700 1 -- start start 2023-12-16T20:49:00.407 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.405+0000 7f38ca4f3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38c40fb1e0 0x7f38c40fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:00.407 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.405+0000 7f38ca4f3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f38c4102c70 0x7f38c40fc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:00.407 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.405+0000 7f38ca4f3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c40fcf00 con 0x7f38c4102c70 2023-12-16T20:49:00.407 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.405+0000 7f38ca4f3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c40fd070 con 0x7f38c40fb1e0 2023-12-16T20:49:00.407 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.405+0000 7f38c37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f38c4102c70 0x7f38c40fc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:00.407 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.405+0000 7f38c37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f38c4102c70 0x7f38c40fc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35554/0 (socket says 172.21.15.52:35554) 2023-12-16T20:49:00.408 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.405+0000 7f38c37fe700 1 -- 172.21.15.52:0/1292985232 learned_addr learned my addr 172.21.15.52:0/1292985232 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:00.408 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.405+0000 7f38c3fff700 1 --2- 172.21.15.52:0/1292985232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38c40fb1e0 0x7f38c40fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:00.408 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.406+0000 7f38c37fe700 1 -- 172.21.15.52:0/1292985232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38c40fb1e0 msgr2=0x7f38c40fc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:00.408 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.406+0000 7f38c37fe700 1 --2- 172.21.15.52:0/1292985232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38c40fb1e0 0x7f38c40fc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.408 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.406+0000 7f38c37fe700 1 -- 172.21.15.52:0/1292985232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b4005040 con 0x7f38c4102c70 2023-12-16T20:49:00.408 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.406+0000 7f38c37fe700 1 --2- 172.21.15.52:0/1292985232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f38c4102c70 0x7f38c40fc900 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f38b8002a50 tx=0x7f38b80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:00.408 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.406+0000 7f38c0ff9700 1 -- 172.21.15.52:0/1292985232 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38b8007cc0 con 0x7f38c4102c70 2023-12-16T20:49:00.408 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.406+0000 7f38c0ff9700 1 -- 172.21.15.52:0/1292985232 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38b800f040 con 0x7f38c4102c70 2023-12-16T20:49:00.408 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.406+0000 7f38c0ff9700 1 -- 172.21.15.52:0/1292985232 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38b800d9c0 con 0x7f38c4102c70 2023-12-16T20:49:00.408 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.406+0000 7f38ca4f3700 1 -- 172.21.15.52:0/1292985232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c4119840 con 0x7f38c4102c70 2023-12-16T20:49:00.408 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.406+0000 7f38ca4f3700 1 -- 172.21.15.52:0/1292985232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c4119d20 con 0x7f38c4102c70 2023-12-16T20:49:00.409 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.407+0000 7f38ca4f3700 1 -- 172.21.15.52:0/1292985232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c40fff10 con 0x7f38c4102c70 2023-12-16T20:49:00.409 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.408+0000 7f38c0ff9700 1 -- 172.21.15.52:0/1292985232 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f38b800db20 con 0x7f38c4102c70 2023-12-16T20:49:00.409 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.408+0000 7f38c0ff9700 1 --2- 172.21.15.52:0/1292985232 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f38ac068f60 0x7f38ac06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:00.409 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.409+0000 7f38c0ff9700 1 -- 172.21.15.52:0/1292985232 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f38b80866a0 con 0x7f38c4102c70 2023-12-16T20:49:00.409 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.409+0000 7f38c3fff700 1 --2- 172.21.15.52:0/1292985232 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f38ac068f60 0x7f38ac06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:00.410 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.409+0000 7f38c3fff700 1 --2- 172.21.15.52:0/1292985232 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f38ac068f60 0x7f38ac06b410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f38b4004f90 tx=0x7f38b4009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:00.413 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.413+0000 7f38c0ff9700 1 -- 172.21.15.52:0/1292985232 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f38b8052eb0 con 0x7f38c4102c70 2023-12-16T20:49:00.617 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.616+0000 7f38ca4f3700 1 -- 172.21.15.52:0/1292985232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f38c4072cc0 con 0x7f38c4102c70 2023-12-16T20:49:00.617 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.616+0000 7f38c0ff9700 1 -- 172.21.15.52:0/1292985232 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f38b80564d0 con 0x7f38c4102c70 2023-12-16T20:49:00.617 INFO:teuthology.orchestra.run.smithi052.stdout:81604378642 2023-12-16T20:49:00.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.618+0000 7f38ca4f3700 1 -- 172.21.15.52:0/1292985232 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f38ac068f60 msgr2=0x7f38ac06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:00.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.618+0000 7f38ca4f3700 1 --2- 172.21.15.52:0/1292985232 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f38ac068f60 0x7f38ac06b410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f38b4004f90 tx=0x7f38b4009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:00.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.618+0000 7f38ca4f3700 1 -- 172.21.15.52:0/1292985232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f38c4102c70 msgr2=0x7f38c40fc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:00.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.618+0000 7f38ca4f3700 1 --2- 172.21.15.52:0/1292985232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f38c4102c70 0x7f38c40fc900 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f38b8002a50 tx=0x7f38b80046b0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.618+0000 7f38ca4f3700 1 -- 172.21.15.52:0/1292985232 shutdown_connections 2023-12-16T20:49:00.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.618+0000 7f38ca4f3700 1 --2- 172.21.15.52:0/1292985232 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f38ac068f60 0x7f38ac06b410 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.618+0000 7f38ca4f3700 1 --2- 172.21.15.52:0/1292985232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38c40fb1e0 0x7f38c40fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.618+0000 7f38ca4f3700 1 --2- 172.21.15.52:0/1292985232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f38c4102c70 0x7f38c40fc900 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.618+0000 7f38ca4f3700 1 -- 172.21.15.52:0/1292985232 >> 172.21.15.52:0/1292985232 conn(0x7f38c40711a0 msgr2=0x7f38c40fa210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:00.620 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.619+0000 7f38ca4f3700 1 -- 172.21.15.52:0/1292985232 shutdown_connections 2023-12-16T20:49:00.620 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.619+0000 7f38ca4f3700 1 -- 172.21.15.52:0/1292985232 wait complete. 2023-12-16T20:49:00.730 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822664 got 154618822665 for osd.5 2023-12-16T20:49:00.730 DEBUG:teuthology.parallel:result is None 2023-12-16T20:49:00.764 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.476+0000 7f93ccc6b700 1 -- 172.21.15.52:0/1826900143 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93c8101b60 msgr2=0x7f93c8101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:00.765 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.476+0000 7f93ccc6b700 1 --2- 172.21.15.52:0/1826900143 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93c8101b60 0x7f93c8101f30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f93c0009ce0 tx=0x7f93c0004d10 comp rx=0 tx=0).stop 2023-12-16T20:49:00.765 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.476+0000 7f93ccc6b700 1 -- 172.21.15.52:0/1826900143 shutdown_connections 2023-12-16T20:49:00.765 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.476+0000 7f93ccc6b700 1 --2- 172.21.15.52:0/1826900143 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93c80fbb00 0x7f93c80fbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.765 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.476+0000 7f93ccc6b700 1 --2- 172.21.15.52:0/1826900143 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93c8101b60 0x7f93c8101f30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.765 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.476+0000 7f93ccc6b700 1 -- 172.21.15.52:0/1826900143 >> 172.21.15.52:0/1826900143 conn(0x7f93c80f7680 msgr2=0x7f93c80f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:00.765 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.476+0000 7f93ccc6b700 1 -- 172.21.15.52:0/1826900143 shutdown_connections 2023-12-16T20:49:00.765 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.476+0000 7f93ccc6b700 1 -- 172.21.15.52:0/1826900143 wait complete. 2023-12-16T20:49:00.765 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.477+0000 7f93ccc6b700 1 Processor -- start 2023-12-16T20:49:00.765 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.477+0000 7f93ccc6b700 1 -- start start 2023-12-16T20:49:00.765 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.477+0000 7f93ccc6b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93c80fbb00 0x7f93c810e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:00.765 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.477+0000 7f93ccc6b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93c8101b60 0x7f93c810ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:00.765 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.477+0000 7f93ccc6b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c810f3a0 con 0x7f93c80fbb00 2023-12-16T20:49:00.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.477+0000 7f93ccc6b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c810f4e0 con 0x7f93c8101b60 2023-12-16T20:49:00.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.477+0000 7f93c659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93c80fbb00 0x7f93c810e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:00.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.478+0000 7f93c659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93c80fbb00 0x7f93c810e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35564/0 (socket says 172.21.15.52:35564) 2023-12-16T20:49:00.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.478+0000 7f93c659c700 1 -- 172.21.15.52:0/4086342083 learned_addr learned my addr 172.21.15.52:0/4086342083 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:00.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.478+0000 7f93c5d9b700 1 --2- 172.21.15.52:0/4086342083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93c8101b60 0x7f93c810ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:00.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.478+0000 7f93c659c700 1 -- 172.21.15.52:0/4086342083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93c8101b60 msgr2=0x7f93c810ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:00.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.478+0000 7f93c659c700 1 --2- 172.21.15.52:0/4086342083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93c8101b60 0x7f93c810ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.478+0000 7f93c659c700 1 -- 172.21.15.52:0/4086342083 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c0009940 con 0x7f93c80fbb00 2023-12-16T20:49:00.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.478+0000 7f93c659c700 1 --2- 172.21.15.52:0/4086342083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93c80fbb00 0x7f93c810e8d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f93c000bee0 tx=0x7f93c0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:00.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.478+0000 7f93beffd700 1 -- 172.21.15.52:0/4086342083 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c000e040 con 0x7f93c80fbb00 2023-12-16T20:49:00.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.478+0000 7f93ccc6b700 1 -- 172.21.15.52:0/4086342083 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c81177e0 con 0x7f93c80fbb00 2023-12-16T20:49:00.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.478+0000 7f93beffd700 1 -- 172.21.15.52:0/4086342083 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93c0008f00 con 0x7f93c80fbb00 2023-12-16T20:49:00.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.478+0000 7f93beffd700 1 -- 172.21.15.52:0/4086342083 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c001c710 con 0x7f93c80fbb00 2023-12-16T20:49:00.767 INFO:teuthology.orchestra.run.smithi052.stdout:107374182414 2023-12-16T20:49:00.767 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.478+0000 7f93ccc6b700 1 -- 172.21.15.52:0/4086342083 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c8117cf0 con 0x7f93c80fbb00 2023-12-16T20:49:00.767 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.479+0000 7f93ccc6b700 1 -- 172.21.15.52:0/4086342083 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c8103f40 con 0x7f93c80fbb00 2023-12-16T20:49:00.767 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.480+0000 7f93beffd700 1 -- 172.21.15.52:0/4086342083 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f93c0012b50 con 0x7f93c80fbb00 2023-12-16T20:49:00.767 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.480+0000 7f93beffd700 1 --2- 172.21.15.52:0/4086342083 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f93b4068f60 0x7f93b406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:00.767 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.481+0000 7f93beffd700 1 -- 172.21.15.52:0/4086342083 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f93c0088740 con 0x7f93c80fbb00 2023-12-16T20:49:00.767 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.481+0000 7f93c5d9b700 1 --2- 172.21.15.52:0/4086342083 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f93b4068f60 0x7f93b406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:00.767 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.481+0000 7f93c5d9b700 1 --2- 172.21.15.52:0/4086342083 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f93b4068f60 0x7f93b406b410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f93b0007fd0 tx=0x7f93b0009040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:00.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.485+0000 7f93beffd700 1 -- 172.21.15.52:0/4086342083 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f93c0054ed0 con 0x7f93c80fbb00 2023-12-16T20:49:00.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.693+0000 7f93ccc6b700 1 -- 172.21.15.52:0/4086342083 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f93c80fb280 con 0x7f93c80fbb00 2023-12-16T20:49:00.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.693+0000 7f93beffd700 1 -- 172.21.15.52:0/4086342083 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f93c00584f0 con 0x7f93c80fbb00 2023-12-16T20:49:00.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.695+0000 7f93ccc6b700 1 -- 172.21.15.52:0/4086342083 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f93b4068f60 msgr2=0x7f93b406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:00.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.695+0000 7f93ccc6b700 1 --2- 172.21.15.52:0/4086342083 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f93b4068f60 0x7f93b406b410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f93b0007fd0 tx=0x7f93b0009040 comp rx=0 tx=0).stop 2023-12-16T20:49:00.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.695+0000 7f93ccc6b700 1 -- 172.21.15.52:0/4086342083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93c80fbb00 msgr2=0x7f93c810e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:00.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.695+0000 7f93ccc6b700 1 --2- 172.21.15.52:0/4086342083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93c80fbb00 0x7f93c810e8d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f93c000bee0 tx=0x7f93c0004d10 comp rx=0 tx=0).stop 2023-12-16T20:49:00.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.695+0000 7f93ccc6b700 1 -- 172.21.15.52:0/4086342083 shutdown_connections 2023-12-16T20:49:00.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.695+0000 7f93ccc6b700 1 --2- 172.21.15.52:0/4086342083 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f93b4068f60 0x7f93b406b410 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.695+0000 7f93ccc6b700 1 --2- 172.21.15.52:0/4086342083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93c80fbb00 0x7f93c810e8d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.695+0000 7f93ccc6b700 1 --2- 172.21.15.52:0/4086342083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93c8101b60 0x7f93c810ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:00.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.695+0000 7f93ccc6b700 1 -- 172.21.15.52:0/4086342083 >> 172.21.15.52:0/4086342083 conn(0x7f93c80f7680 msgr2=0x7f93c80f8380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:00.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.696+0000 7f93ccc6b700 1 -- 172.21.15.52:0/4086342083 shutdown_connections 2023-12-16T20:49:00.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:00.696+0000 7f93ccc6b700 1 -- 172.21.15.52:0/4086342083 wait complete. 2023-12-16T20:49:01.165 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378641 got 81604378642 for osd.2 2023-12-16T20:49:01.165 DEBUG:teuthology.parallel:result is None 2023-12-16T20:49:01.233 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:00 smithi052 ceph-mon[80761]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:01.234 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:00 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3163676355' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-12-16T20:49:01.234 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:00 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/1292985232' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-12-16T20:49:01.234 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:00 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/4086342083' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-12-16T20:49:01.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:00 smithi150 ceph-mon[88000]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:01.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:00 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/3163676355' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-12-16T20:49:01.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:00 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/1292985232' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-12-16T20:49:01.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:00 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/4086342083' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-12-16T20:49:01.443 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182413 got 107374182414 for osd.3 2023-12-16T20:49:01.444 DEBUG:teuthology.parallel:result is None 2023-12-16T20:49:01.444 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-12-16T20:49:01.444 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph pg dump --format=json 2023-12-16T20:49:01.720 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:02.701 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.700+0000 7fee1920b700 1 -- 172.21.15.52:0/1457988495 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee14102c00 msgr2=0x7fee14102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:02.701 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.700+0000 7fee1920b700 1 --2- 172.21.15.52:0/1457988495 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee14102c00 0x7fee14102fd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fee04004940 tx=0x7fee04009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:02.701 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.700+0000 7fee1920b700 1 -- 172.21.15.52:0/1457988495 shutdown_connections 2023-12-16T20:49:02.701 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.700+0000 7fee1920b700 1 --2- 172.21.15.52:0/1457988495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee141035a0 0x7fee140fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:02.701 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.700+0000 7fee1920b700 1 --2- 172.21.15.52:0/1457988495 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee14102c00 0x7fee14102fd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:02.702 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.700+0000 7fee1920b700 1 -- 172.21.15.52:0/1457988495 >> 172.21.15.52:0/1457988495 conn(0x7fee140f8790 msgr2=0x7fee140faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:02.702 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.700+0000 7fee1920b700 1 -- 172.21.15.52:0/1457988495 shutdown_connections 2023-12-16T20:49:02.702 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.700+0000 7fee1920b700 1 -- 172.21.15.52:0/1457988495 wait complete. 2023-12-16T20:49:02.703 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.701+0000 7fee1920b700 1 Processor -- start 2023-12-16T20:49:02.703 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.701+0000 7fee1920b700 1 -- start start 2023-12-16T20:49:02.703 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.701+0000 7fee1920b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee14102c00 0x7fee141136b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:02.703 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.701+0000 7fee1920b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee141035a0 0x7fee14113c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:02.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.701+0000 7fee1920b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1410d570 con 0x7fee14102c00 2023-12-16T20:49:02.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.701+0000 7fee1920b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1410d6e0 con 0x7fee141035a0 2023-12-16T20:49:02.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.701+0000 7fee12d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee14102c00 0x7fee141136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:02.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.701+0000 7fee12d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee14102c00 0x7fee141136b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46042/0 (socket says 172.21.15.52:46042) 2023-12-16T20:49:02.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.701+0000 7fee12d9d700 1 -- 172.21.15.52:0/3646980013 learned_addr learned my addr 172.21.15.52:0/3646980013 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:02.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.701+0000 7fee1259c700 1 --2- 172.21.15.52:0/3646980013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee141035a0 0x7fee14113c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:02.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.702+0000 7fee12d9d700 1 -- 172.21.15.52:0/3646980013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee141035a0 msgr2=0x7fee14113c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:02.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.702+0000 7fee12d9d700 1 --2- 172.21.15.52:0/3646980013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee141035a0 0x7fee14113c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:02.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.702+0000 7fee12d9d700 1 -- 172.21.15.52:0/3646980013 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee04005040 con 0x7fee14102c00 2023-12-16T20:49:02.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.702+0000 7fee1259c700 1 --2- 172.21.15.52:0/3646980013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee141035a0 0x7fee14113c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T20:49:02.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.702+0000 7fee12d9d700 1 --2- 172.21.15.52:0/3646980013 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee14102c00 0x7fee141136b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fee04009fd0 tx=0x7fee04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:02.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.702+0000 7fedfb7fe700 1 -- 172.21.15.52:0/3646980013 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee0400e040 con 0x7fee14102c00 2023-12-16T20:49:02.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.702+0000 7fedfb7fe700 1 -- 172.21.15.52:0/3646980013 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee0400bc50 con 0x7fee14102c00 2023-12-16T20:49:02.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.702+0000 7fee1920b700 1 -- 172.21.15.52:0/3646980013 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1410d960 con 0x7fee14102c00 2023-12-16T20:49:02.705 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.702+0000 7fedfb7fe700 1 -- 172.21.15.52:0/3646980013 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee04013ea0 con 0x7fee14102c00 2023-12-16T20:49:02.705 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.702+0000 7fee1920b700 1 -- 172.21.15.52:0/3646980013 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1410dea0 con 0x7fee14102c00 2023-12-16T20:49:02.706 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.703+0000 7fee1920b700 1 -- 172.21.15.52:0/3646980013 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee140fff10 con 0x7fee14102c00 2023-12-16T20:49:02.706 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.704+0000 7fedfb7fe700 1 -- 172.21.15.52:0/3646980013 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fee04013420 con 0x7fee14102c00 2023-12-16T20:49:02.707 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.704+0000 7fedfb7fe700 1 --2- 172.21.15.52:0/3646980013 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fedfc068f70 0x7fedfc06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:02.707 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.704+0000 7fedfb7fe700 1 -- 172.21.15.52:0/3646980013 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fee04087fc0 con 0x7fee14102c00 2023-12-16T20:49:02.707 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.704+0000 7fee1259c700 1 --2- 172.21.15.52:0/3646980013 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fedfc068f70 0x7fedfc06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:02.707 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.704+0000 7fee1259c700 1 --2- 172.21.15.52:0/3646980013 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fedfc068f70 0x7fedfc06b420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fee1410ef50 tx=0x7fee08008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:02.709 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.708+0000 7fedfb7fe700 1 -- 172.21.15.52:0/3646980013 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fee040581e0 con 0x7fee14102c00 2023-12-16T20:49:02.902 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.901+0000 7fee1920b700 1 -- 172.21.15.52:0/3646980013 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fee1404bb80 con 0x7fedfc068f70 2023-12-16T20:49:02.903 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.903+0000 7fedfb7fe700 1 -- 172.21.15.52:0/3646980013 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+21822 (secure 0 0 0) 0x7fee1404bb80 con 0x7fedfc068f70 2023-12-16T20:49:02.904 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:49:02.906 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.905+0000 7fee1920b700 1 -- 172.21.15.52:0/3646980013 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fedfc068f70 msgr2=0x7fedfc06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:02.906 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.905+0000 7fee1920b700 1 --2- 172.21.15.52:0/3646980013 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fedfc068f70 0x7fedfc06b420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fee1410ef50 tx=0x7fee08008040 comp rx=0 tx=0).stop 2023-12-16T20:49:02.906 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.905+0000 7fee1920b700 1 -- 172.21.15.52:0/3646980013 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee14102c00 msgr2=0x7fee141136b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:02.906 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.905+0000 7fee1920b700 1 --2- 172.21.15.52:0/3646980013 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee14102c00 0x7fee141136b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fee04009fd0 tx=0x7fee04009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:02.906 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.905+0000 7fee1920b700 1 -- 172.21.15.52:0/3646980013 shutdown_connections 2023-12-16T20:49:02.906 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.905+0000 7fee1920b700 1 --2- 172.21.15.52:0/3646980013 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fedfc068f70 0x7fedfc06b420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:02.906 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.905+0000 7fee1920b700 1 --2- 172.21.15.52:0/3646980013 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee14102c00 0x7fee141136b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:02.906 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.905+0000 7fee1920b700 1 --2- 172.21.15.52:0/3646980013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee141035a0 0x7fee14113c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:02.906 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.905+0000 7fee1920b700 1 -- 172.21.15.52:0/3646980013 >> 172.21.15.52:0/3646980013 conn(0x7fee140f8790 msgr2=0x7fee140fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:02.906 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.905+0000 7fee1920b700 1 -- 172.21.15.52:0/3646980013 shutdown_connections 2023-12-16T20:49:02.906 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:02.905+0000 7fee1920b700 1 -- 172.21.15.52:0/3646980013 wait complete. 2023-12-16T20:49:02.907 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2023-12-16T20:49:03.239 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:02 smithi052 ceph-mon[80761]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:03.239 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:02 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:49:03.265 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2023-12-16T20:49:01.915131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787432,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683480,"statfs":{"total":575970213888,"available":574139883520,"internally_reserved":0,"allocated":3792896,"data_stored":2538747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002006"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":187,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T20:48:33.952664+0000","last_change":"2023-12-16T20:48:16.985263+0000","last_active":"2023-12-16T20:48:33.952664+0000","last_peered":"2023-12-16T20:48:33.952664+0000","last_clean":"2023-12-16T20:48:33.952664+0000","last_became_active":"2023-12-16T20:48:16.984990+0000","last_became_peered":"2023-12-16T20:48:16.984990+0000","last_unstale":"2023-12-16T20:48:33.952664+0000","last_undegraded":"2023-12-16T20:48:33.952664+0000","last_fullsized":"2023-12-16T20:48:33.952664+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T20:47:46.399166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T20:47:46.399166+0000","last_clean_scrub_stamp":"2023-12-16T20:47:46.399166+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:28:40.077518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":36,"seq":154618822665,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297616,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447536,"statfs":{"total":95995035648,"available":95690276864,"internally_reserved":0,"allocated":335872,"data_stored":127992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953484,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297616,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447536,"statfs":{"total":95995035648,"available":95690276864,"internally_reserved":0,"allocated":335872,"data_stored":127992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182414,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298192,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446960,"statfs":{"total":95995035648,"available":95689687040,"internally_reserved":0,"allocated":925696,"data_stored":718051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297616,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447536,"statfs":{"total":95995035648,"available":95690276864,"internally_reserved":0,"allocated":335872,"data_stored":127992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 20:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.33900000000000002}]},{"osd":1,"last update":"Sat Dec 16 20:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.315},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.35899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705689,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446956,"statfs":{"total":95995035648,"available":95689682944,"internally_reserved":0,"allocated":929792,"data_stored":718360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 20:48:30 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Dec 16 20:48:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.317}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542166,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446956,"statfs":{"total":95995035648,"available":95689682944,"internally_reserved":0,"allocated":929792,"data_stored":718360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 20:48:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.248}]},{"osd":2,"last update":"Sat Dec 16 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.28499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T20:49:03.266 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph pg dump --format=json 2023-12-16T20:49:03.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:02 smithi150 ceph-mon[88000]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:03.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:02 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:49:03.533 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:04.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:03 smithi052 ceph-mon[80761]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T20:49:04.339 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:03 smithi150 ceph-mon[88000]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T20:49:04.517 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.514+0000 7fd65abfd700 1 -- 172.21.15.52:0/2973302723 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6541035a0 msgr2=0x7fd6540fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:04.517 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.514+0000 7fd65abfd700 1 --2- 172.21.15.52:0/2973302723 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6541035a0 0x7fd6540fcc10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd640004940 tx=0x7fd640009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:04.517 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.515+0000 7fd65abfd700 1 -- 172.21.15.52:0/2973302723 shutdown_connections 2023-12-16T20:49:04.517 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.515+0000 7fd65abfd700 1 --2- 172.21.15.52:0/2973302723 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6541035a0 0x7fd6540fcc10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:04.517 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.515+0000 7fd65abfd700 1 --2- 172.21.15.52:0/2973302723 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd654102c00 0x7fd654102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:04.517 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.515+0000 7fd65abfd700 1 -- 172.21.15.52:0/2973302723 >> 172.21.15.52:0/2973302723 conn(0x7fd6540f8790 msgr2=0x7fd6540faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:04.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.515+0000 7fd65abfd700 1 -- 172.21.15.52:0/2973302723 shutdown_connections 2023-12-16T20:49:04.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.515+0000 7fd65abfd700 1 -- 172.21.15.52:0/2973302723 wait complete. 2023-12-16T20:49:04.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.515+0000 7fd65abfd700 1 Processor -- start 2023-12-16T20:49:04.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.515+0000 7fd65abfd700 1 -- start start 2023-12-16T20:49:04.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd65abfd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd654102c00 0x7fd65406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:04.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd65abfd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6541035a0 0x7fd654070480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:04.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd65abfd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd654070a80 con 0x7fd6541035a0 2023-12-16T20:49:04.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd65abfd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd654070bf0 con 0x7fd654102c00 2023-12-16T20:49:04.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd653fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6541035a0 0x7fd654070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:04.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd653fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6541035a0 0x7fd654070480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46066/0 (socket says 172.21.15.52:46066) 2023-12-16T20:49:04.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd653fff700 1 -- 172.21.15.52:0/4237765523 learned_addr learned my addr 172.21.15.52:0/4237765523 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:04.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd658999700 1 --2- 172.21.15.52:0/4237765523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd654102c00 0x7fd65406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:04.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd653fff700 1 -- 172.21.15.52:0/4237765523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd654102c00 msgr2=0x7fd65406ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:04.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd653fff700 1 --2- 172.21.15.52:0/4237765523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd654102c00 0x7fd65406ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:04.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd653fff700 1 -- 172.21.15.52:0/4237765523 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd640005040 con 0x7fd6541035a0 2023-12-16T20:49:04.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd653fff700 1 --2- 172.21.15.52:0/4237765523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6541035a0 0x7fd654070480 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd640000c00 tx=0x7fd640009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:04.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd6517fa700 1 -- 172.21.15.52:0/4237765523 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd640021070 con 0x7fd6541035a0 2023-12-16T20:49:04.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd6517fa700 1 -- 172.21.15.52:0/4237765523 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6400093b0 con 0x7fd6541035a0 2023-12-16T20:49:04.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd65abfd700 1 -- 172.21.15.52:0/4237765523 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6541197f0 con 0x7fd6541035a0 2023-12-16T20:49:04.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd6517fa700 1 -- 172.21.15.52:0/4237765523 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd64000e070 con 0x7fd6541035a0 2023-12-16T20:49:04.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.516+0000 7fd65abfd700 1 -- 172.21.15.52:0/4237765523 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd654119bf0 con 0x7fd6541035a0 2023-12-16T20:49:04.520 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.518+0000 7fd65abfd700 1 -- 172.21.15.52:0/4237765523 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6541050d0 con 0x7fd6541035a0 2023-12-16T20:49:04.520 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.518+0000 7fd6517fa700 1 -- 172.21.15.52:0/4237765523 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fd64001c670 con 0x7fd6541035a0 2023-12-16T20:49:04.520 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.518+0000 7fd6517fa700 1 --2- 172.21.15.52:0/4237765523 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd644068f60 0x7fd64406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:04.520 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.518+0000 7fd6517fa700 1 -- 172.21.15.52:0/4237765523 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd640087be0 con 0x7fd6541035a0 2023-12-16T20:49:04.520 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.518+0000 7fd658999700 1 --2- 172.21.15.52:0/4237765523 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd644068f60 0x7fd64406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:04.520 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.519+0000 7fd658999700 1 --2- 172.21.15.52:0/4237765523 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd644068f60 0x7fd64406b410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd648009b80 tx=0x7fd64800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:04.523 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.523+0000 7fd6517fa700 1 -- 172.21.15.52:0/4237765523 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd6400543f0 con 0x7fd6541035a0 2023-12-16T20:49:04.713 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.713+0000 7fd65abfd700 1 -- 172.21.15.52:0/4237765523 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd654071680 con 0x7fd644068f60 2023-12-16T20:49:04.715 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.714+0000 7fd6517fa700 1 -- 172.21.15.52:0/4237765523 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+22307 (secure 0 0 0) 0x7fd654071680 con 0x7fd644068f60 2023-12-16T20:49:04.715 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:49:04.716 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.716+0000 7fd65abfd700 1 -- 172.21.15.52:0/4237765523 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd644068f60 msgr2=0x7fd64406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:04.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.716+0000 7fd65abfd700 1 --2- 172.21.15.52:0/4237765523 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd644068f60 0x7fd64406b410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd648009b80 tx=0x7fd64800b040 comp rx=0 tx=0).stop 2023-12-16T20:49:04.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.716+0000 7fd65abfd700 1 -- 172.21.15.52:0/4237765523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6541035a0 msgr2=0x7fd654070480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:04.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.716+0000 7fd65abfd700 1 --2- 172.21.15.52:0/4237765523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6541035a0 0x7fd654070480 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd640000c00 tx=0x7fd640009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:04.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.716+0000 7fd65abfd700 1 -- 172.21.15.52:0/4237765523 shutdown_connections 2023-12-16T20:49:04.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.716+0000 7fd65abfd700 1 --2- 172.21.15.52:0/4237765523 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd644068f60 0x7fd64406b410 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:04.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.716+0000 7fd65abfd700 1 --2- 172.21.15.52:0/4237765523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd654102c00 0x7fd65406ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:04.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.716+0000 7fd65abfd700 1 --2- 172.21.15.52:0/4237765523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6541035a0 0x7fd654070480 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:04.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.716+0000 7fd65abfd700 1 -- 172.21.15.52:0/4237765523 >> 172.21.15.52:0/4237765523 conn(0x7fd6540f8790 msgr2=0x7fd6540fa220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:04.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.716+0000 7fd65abfd700 1 -- 172.21.15.52:0/4237765523 shutdown_connections 2023-12-16T20:49:04.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:04.716+0000 7fd65abfd700 1 -- 172.21.15.52:0/4237765523 wait complete. 2023-12-16T20:49:04.718 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2023-12-16T20:49:05.048 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2023-12-16T20:49:03.915452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787432,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683480,"statfs":{"total":575970213888,"available":574139883520,"internally_reserved":0,"allocated":3792896,"data_stored":2538747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001987"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":187,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T20:48:33.952664+0000","last_change":"2023-12-16T20:48:16.985263+0000","last_active":"2023-12-16T20:48:33.952664+0000","last_peered":"2023-12-16T20:48:33.952664+0000","last_clean":"2023-12-16T20:48:33.952664+0000","last_became_active":"2023-12-16T20:48:16.984990+0000","last_became_peered":"2023-12-16T20:48:16.984990+0000","last_unstale":"2023-12-16T20:48:33.952664+0000","last_undegraded":"2023-12-16T20:48:33.952664+0000","last_fullsized":"2023-12-16T20:48:33.952664+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T20:47:46.399166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T20:47:46.399166+0000","last_clean_scrub_stamp":"2023-12-16T20:47:46.399166+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:28:40.077518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":36,"seq":154618822665,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297616,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447536,"statfs":{"total":95995035648,"available":95690276864,"internally_reserved":0,"allocated":335872,"data_stored":127992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953485,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297616,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447536,"statfs":{"total":95995035648,"available":95690276864,"internally_reserved":0,"allocated":335872,"data_stored":127992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182415,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298192,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446960,"statfs":{"total":95995035648,"available":95689687040,"internally_reserved":0,"allocated":925696,"data_stored":718051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297616,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447536,"statfs":{"total":95995035648,"available":95690276864,"internally_reserved":0,"allocated":335872,"data_stored":127992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 20:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.33900000000000002}]},{"osd":1,"last update":"Sat Dec 16 20:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.315},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.35899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705689,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446956,"statfs":{"total":95995035648,"available":95689682944,"internally_reserved":0,"allocated":929792,"data_stored":718360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 20:48:30 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Dec 16 20:48:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.317}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542167,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446956,"statfs":{"total":95995035648,"available":95689682944,"internally_reserved":0,"allocated":929792,"data_stored":718360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 20:48:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.32700000000000001}]},{"osd":2,"last update":"Sat Dec 16 20:48:49 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.251}]},{"osd":3,"last update":"Sat Dec 16 20:49:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.51800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T20:49:05.049 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-12-16T20:49:05.049 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-16T20:49:05.049 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-12-16T20:49:05.049 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-16T20:49:05.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:04 smithi052 ceph-mon[80761]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:05.318 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:05.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:04 smithi150 ceph-mon[88000]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:06.084 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:05 smithi052 ceph-mon[80761]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T20:49:06.238 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.237+0000 7f01fc235700 1 -- 172.21.15.52:0/3184776584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01f4101b60 msgr2=0x7f01f4101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:06.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.237+0000 7f01fc235700 1 --2- 172.21.15.52:0/3184776584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01f4101b60 0x7f01f4101f30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009ce0 tx=0x7f01e4004d10 comp rx=0 tx=0).stop 2023-12-16T20:49:06.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.237+0000 7f01fc235700 1 -- 172.21.15.52:0/3184776584 shutdown_connections 2023-12-16T20:49:06.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.237+0000 7f01fc235700 1 --2- 172.21.15.52:0/3184776584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f40fbb00 0x7f01f40fbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:06.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.237+0000 7f01fc235700 1 --2- 172.21.15.52:0/3184776584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01f4101b60 0x7f01f4101f30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:06.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.237+0000 7f01fc235700 1 -- 172.21.15.52:0/3184776584 >> 172.21.15.52:0/3184776584 conn(0x7f01f40f7680 msgr2=0x7f01f40f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:06.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.237+0000 7f01fc235700 1 -- 172.21.15.52:0/3184776584 shutdown_connections 2023-12-16T20:49:06.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.237+0000 7f01fc235700 1 -- 172.21.15.52:0/3184776584 wait complete. 2023-12-16T20:49:06.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.238+0000 7f01fc235700 1 Processor -- start 2023-12-16T20:49:06.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.238+0000 7f01fc235700 1 -- start start 2023-12-16T20:49:06.240 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.238+0000 7f01fc235700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f40fbb00 0x7f01f41137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:06.240 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.238+0000 7f01fc235700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01f4101b60 0x7f01f4113d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:06.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.238+0000 7f01fc235700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f4114400 con 0x7f01f4101b60 2023-12-16T20:49:06.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.238+0000 7f01fc235700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f410d6a0 con 0x7f01f40fbb00 2023-12-16T20:49:06.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.238+0000 7f01f97d0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01f4101b60 0x7f01f4113d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:06.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.239+0000 7f01f97d0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01f4101b60 0x7f01f4113d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46080/0 (socket says 172.21.15.52:46080) 2023-12-16T20:49:06.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.239+0000 7f01f97d0700 1 -- 172.21.15.52:0/2778935741 learned_addr learned my addr 172.21.15.52:0/2778935741 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:06.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.239+0000 7f01f9fd1700 1 --2- 172.21.15.52:0/2778935741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f40fbb00 0x7f01f41137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:06.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.239+0000 7f01f97d0700 1 -- 172.21.15.52:0/2778935741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f40fbb00 msgr2=0x7f01f41137e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:06.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.239+0000 7f01f97d0700 1 --2- 172.21.15.52:0/2778935741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f40fbb00 0x7f01f41137e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:06.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.239+0000 7f01f97d0700 1 -- 172.21.15.52:0/2778935741 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e4009940 con 0x7f01f4101b60 2023-12-16T20:49:06.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.239+0000 7f01f97d0700 1 --2- 172.21.15.52:0/2778935741 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01f4101b60 0x7f01f4113d20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f01e800b790 tx=0x7f01e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:06.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.239+0000 7f01f27fc700 1 -- 172.21.15.52:0/2778935741 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01e800f6f0 con 0x7f01f4101b60 2023-12-16T20:49:06.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.239+0000 7f01f27fc700 1 -- 172.21.15.52:0/2778935741 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f01e800fd00 con 0x7f01f4101b60 2023-12-16T20:49:06.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.239+0000 7f01f27fc700 1 -- 172.21.15.52:0/2778935741 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01e80103a0 con 0x7f01f4101b60 2023-12-16T20:49:06.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.239+0000 7f01fc235700 1 -- 172.21.15.52:0/2778935741 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f410d980 con 0x7f01f4101b60 2023-12-16T20:49:06.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.239+0000 7f01fc235700 1 -- 172.21.15.52:0/2778935741 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f410dec0 con 0x7f01f4101b60 2023-12-16T20:49:06.242 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.240+0000 7f01fc235700 1 -- 172.21.15.52:0/2778935741 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f4103fc0 con 0x7f01f4101b60 2023-12-16T20:49:06.242 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.241+0000 7f01f27fc700 1 -- 172.21.15.52:0/2778935741 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f01e800f850 con 0x7f01f4101b60 2023-12-16T20:49:06.242 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.241+0000 7f01f27fc700 1 --2- 172.21.15.52:0/2778935741 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f01dc068f70 0x7f01dc06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:06.243 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.241+0000 7f01f9fd1700 1 --2- 172.21.15.52:0/2778935741 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f01dc068f70 0x7f01dc06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:06.243 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.241+0000 7f01f27fc700 1 -- 172.21.15.52:0/2778935741 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f01e8086870 con 0x7f01f4101b60 2023-12-16T20:49:06.243 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.242+0000 7f01f9fd1700 1 --2- 172.21.15.52:0/2778935741 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f01dc068f70 0x7f01dc06b420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f01e4005010 tx=0x7f01e4004d10 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:06.246 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.246+0000 7f01f27fc700 1 -- 172.21.15.52:0/2778935741 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f01e8056b10 con 0x7f01f4101b60 2023-12-16T20:49:06.253 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:05 smithi150 ceph-mon[88000]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T20:49:06.487 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.486+0000 7f01fc235700 1 -- 172.21.15.52:0/2778935741 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f01f40fb280 con 0x7f01f4101b60 2023-12-16T20:49:06.488 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.487+0000 7f01f27fc700 1 -- 172.21.15.52:0/2778935741 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f01e80566a0 con 0x7f01f4101b60 2023-12-16T20:49:06.488 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:49:06.488 INFO:teuthology.orchestra.run.smithi052.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-16T20:49:06.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.489+0000 7f01fc235700 1 -- 172.21.15.52:0/2778935741 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f01dc068f70 msgr2=0x7f01dc06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:06.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.489+0000 7f01fc235700 1 --2- 172.21.15.52:0/2778935741 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f01dc068f70 0x7f01dc06b420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f01e4005010 tx=0x7f01e4004d10 comp rx=0 tx=0).stop 2023-12-16T20:49:06.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.489+0000 7f01fc235700 1 -- 172.21.15.52:0/2778935741 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01f4101b60 msgr2=0x7f01f4113d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:06.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.489+0000 7f01fc235700 1 --2- 172.21.15.52:0/2778935741 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01f4101b60 0x7f01f4113d20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f01e800b790 tx=0x7f01e8009250 comp rx=0 tx=0).stop 2023-12-16T20:49:06.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.489+0000 7f01fc235700 1 -- 172.21.15.52:0/2778935741 shutdown_connections 2023-12-16T20:49:06.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.489+0000 7f01fc235700 1 --2- 172.21.15.52:0/2778935741 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f01dc068f70 0x7f01dc06b420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:06.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.489+0000 7f01fc235700 1 --2- 172.21.15.52:0/2778935741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f40fbb00 0x7f01f41137e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:06.491 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.489+0000 7f01fc235700 1 --2- 172.21.15.52:0/2778935741 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01f4101b60 0x7f01f4113d20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:06.491 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.489+0000 7f01fc235700 1 -- 172.21.15.52:0/2778935741 >> 172.21.15.52:0/2778935741 conn(0x7f01f40f7680 msgr2=0x7f01f40f81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:06.491 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.489+0000 7f01fc235700 1 -- 172.21.15.52:0/2778935741 shutdown_connections 2023-12-16T20:49:06.491 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:06.489+0000 7f01fc235700 1 -- 172.21.15.52:0/2778935741 wait complete. 2023-12-16T20:49:06.833 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-12-16T20:49:06.833 INFO:tasks.cephadm:Setup complete, yielding 2023-12-16T20:49:06.833 INFO:teuthology.run_tasks:Running task print... 2023-12-16T20:49:06.843 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2023-12-16T20:49:06.843 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T20:49:06.856 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi052.front.sepia.ceph.com 2023-12-16T20:49:06.857 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2023-12-16T20:49:07.124 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:07.214 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:06 smithi052 ceph-mon[80761]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:07.214 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:06 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/2778935741' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-16T20:49:07.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:06 smithi150 ceph-mon[88000]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:07.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:06 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/2778935741' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-16T20:49:07.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.995+0000 7f8cfd779700 1 -- 172.21.15.52:0/1740586643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8cf80fcd00 msgr2=0x7f8cf80fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:07.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.995+0000 7f8cfd779700 1 --2- 172.21.15.52:0/1740586643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8cf80fcd00 0x7f8cf80fd150 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8cec004940 tx=0x7f8cec009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:07.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.995+0000 7f8cfd779700 1 -- 172.21.15.52:0/1740586643 shutdown_connections 2023-12-16T20:49:07.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.995+0000 7f8cfd779700 1 --2- 172.21.15.52:0/1740586643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8cf80fcd00 0x7f8cf80fd150 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:07.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.995+0000 7f8cfd779700 1 --2- 172.21.15.52:0/1740586643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cf80fbaf0 0x7f8cf80fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:07.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.995+0000 7f8cfd779700 1 -- 172.21.15.52:0/1740586643 >> 172.21.15.52:0/1740586643 conn(0x7f8cf80f7190 msgr2=0x7f8cf80f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:07.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.995+0000 7f8cfd779700 1 -- 172.21.15.52:0/1740586643 shutdown_connections 2023-12-16T20:49:07.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.995+0000 7f8cfd779700 1 -- 172.21.15.52:0/1740586643 wait complete. 2023-12-16T20:49:07.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.996+0000 7f8cfd779700 1 Processor -- start 2023-12-16T20:49:07.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.996+0000 7f8cfd779700 1 -- start start 2023-12-16T20:49:07.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.996+0000 7f8cfd779700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8cf80fbaf0 0x7f8cf810c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:07.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.996+0000 7f8cfd779700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cf80fcd00 0x7f8cf810c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:07.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.996+0000 7f8cfd779700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf810d040 con 0x7f8cf80fbaf0 2023-12-16T20:49:07.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.996+0000 7f8cfd779700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf810d880 con 0x7f8cf80fcd00 2023-12-16T20:49:07.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.996+0000 7f8cf6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8cf80fbaf0 0x7f8cf810c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:07.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.996+0000 7f8cf6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8cf80fbaf0 0x7f8cf810c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46104/0 (socket says 172.21.15.52:46104) 2023-12-16T20:49:07.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.996+0000 7f8cf6ffd700 1 -- 172.21.15.52:0/3610297192 learned_addr learned my addr 172.21.15.52:0/3610297192 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:07.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.996+0000 7f8cf6ffd700 1 -- 172.21.15.52:0/3610297192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cf80fcd00 msgr2=0x7f8cf810c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:07.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.996+0000 7f8cf6ffd700 1 --2- 172.21.15.52:0/3610297192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cf80fcd00 0x7f8cf810c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:07.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.996+0000 7f8cf6ffd700 1 -- 172.21.15.52:0/3610297192 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec005040 con 0x7f8cf80fbaf0 2023-12-16T20:49:07.999 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.996+0000 7f8cf6ffd700 1 --2- 172.21.15.52:0/3610297192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8cf80fbaf0 0x7f8cf810c470 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8ce800b790 tx=0x7f8ce8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:07.999 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.997+0000 7f8cdf7fe700 1 -- 172.21.15.52:0/3610297192 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ce800f6f0 con 0x7f8cf80fbaf0 2023-12-16T20:49:08.000 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.997+0000 7f8cdf7fe700 1 -- 172.21.15.52:0/3610297192 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ce800fd00 con 0x7f8cf80fbaf0 2023-12-16T20:49:08.000 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.997+0000 7f8cdf7fe700 1 -- 172.21.15.52:0/3610297192 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ce80103a0 con 0x7f8cf80fbaf0 2023-12-16T20:49:08.000 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.997+0000 7f8cfd779700 1 -- 172.21.15.52:0/3610297192 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf810db60 con 0x7f8cf80fbaf0 2023-12-16T20:49:08.000 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.997+0000 7f8cfd779700 1 -- 172.21.15.52:0/3610297192 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf810e0a0 con 0x7f8cf80fbaf0 2023-12-16T20:49:08.000 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.998+0000 7f8cfd779700 1 -- 172.21.15.52:0/3610297192 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf804bb80 con 0x7f8cf80fbaf0 2023-12-16T20:49:08.000 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.998+0000 7f8cdf7fe700 1 -- 172.21.15.52:0/3610297192 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f8ce800f850 con 0x7f8cf80fbaf0 2023-12-16T20:49:08.000 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.998+0000 7f8cdf7fe700 1 --2- 172.21.15.52:0/3610297192 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ce0068f60 0x7f8ce006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:08.000 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.999+0000 7f8cdf7fe700 1 -- 172.21.15.52:0/3610297192 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8ce80868b0 con 0x7f8cf80fbaf0 2023-12-16T20:49:08.000 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.999+0000 7f8cf67fc700 1 --2- 172.21.15.52:0/3610297192 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ce0068f60 0x7f8ce006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:08.000 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:07.999+0000 7f8cf67fc700 1 --2- 172.21.15.52:0/3610297192 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ce0068f60 0x7f8ce006b410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8cec00b3b0 tx=0x7f8cec009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:08.004 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:08.003+0000 7f8cdf7fe700 1 -- 172.21.15.52:0/3610297192 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8ce8056ad0 con 0x7f8cf80fbaf0 2023-12-16T20:49:08.201 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:08.200+0000 7f8cfd779700 1 -- 172.21.15.52:0/3610297192 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f8cf8104530 con 0x7f8cf80fbaf0 2023-12-16T20:49:08.205 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:08.204+0000 7f8cdf7fe700 1 -- 172.21.15.52:0/3610297192 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f8ce8056660 con 0x7f8cf80fbaf0 2023-12-16T20:49:08.208 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:08.206+0000 7f8cfd779700 1 -- 172.21.15.52:0/3610297192 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ce0068f60 msgr2=0x7f8ce006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:08.208 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:08.207+0000 7f8cfd779700 1 --2- 172.21.15.52:0/3610297192 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ce0068f60 0x7f8ce006b410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8cec00b3b0 tx=0x7f8cec009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:08.208 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:08.207+0000 7f8cfd779700 1 -- 172.21.15.52:0/3610297192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8cf80fbaf0 msgr2=0x7f8cf810c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:08.208 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:08.207+0000 7f8cfd779700 1 --2- 172.21.15.52:0/3610297192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8cf80fbaf0 0x7f8cf810c470 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8ce800b790 tx=0x7f8ce8009220 comp rx=0 tx=0).stop 2023-12-16T20:49:08.209 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:08.207+0000 7f8cfd779700 1 -- 172.21.15.52:0/3610297192 shutdown_connections 2023-12-16T20:49:08.209 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:08.207+0000 7f8cfd779700 1 --2- 172.21.15.52:0/3610297192 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ce0068f60 0x7f8ce006b410 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:08.209 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:08.207+0000 7f8cfd779700 1 --2- 172.21.15.52:0/3610297192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8cf80fbaf0 0x7f8cf810c470 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:08.209 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:08.207+0000 7f8cfd779700 1 --2- 172.21.15.52:0/3610297192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cf80fcd00 0x7f8cf810c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:08.209 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:08.207+0000 7f8cfd779700 1 -- 172.21.15.52:0/3610297192 >> 172.21.15.52:0/3610297192 conn(0x7f8cf80f7190 msgr2=0x7f8cf80fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:08.209 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:08.207+0000 7f8cfd779700 1 -- 172.21.15.52:0/3610297192 shutdown_connections 2023-12-16T20:49:08.210 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:08.207+0000 7f8cfd779700 1 -- 172.21.15.52:0/3610297192 wait complete. 2023-12-16T20:49:08.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:08 smithi052 ceph-mon[80761]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:08.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:08 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3610297192' entity='client.admin' 2023-12-16T20:49:08.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:08 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:49:08.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:08 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:08.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:08 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:49:08.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:08 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:08.573 INFO:teuthology.run_tasks:Running task print... 2023-12-16T20:49:08.582 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2023-12-16T20:49:08.582 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T20:49:08.594 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi052.front.sepia.ceph.com 2023-12-16T20:49:08.594 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- bash -c 'ceph orch status' 2023-12-16T20:49:08.861 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:08.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:08 smithi150 ceph-mon[88000]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:08.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:08 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/3610297192' entity='client.admin' 2023-12-16T20:49:08.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:08 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:49:08.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:08 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:08.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:08 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:49:08.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:08 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:09.749 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.747+0000 7fd489415700 1 -- 172.21.15.52:0/1825127452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4840fee00 msgr2=0x7fd4840ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:09.749 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.747+0000 7fd489415700 1 --2- 172.21.15.52:0/1825127452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4840fee00 0x7fd4840ff210 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd47c009ce0 tx=0x7fd47c004d10 comp rx=0 tx=0).stop 2023-12-16T20:49:09.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.747+0000 7fd489415700 1 -- 172.21.15.52:0/1825127452 shutdown_connections 2023-12-16T20:49:09.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.747+0000 7fd489415700 1 --2- 172.21.15.52:0/1825127452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4840fff70 0x7fd4841003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:09.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.747+0000 7fd489415700 1 --2- 172.21.15.52:0/1825127452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4840fee00 0x7fd4840ff210 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:09.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.747+0000 7fd489415700 1 -- 172.21.15.52:0/1825127452 >> 172.21.15.52:0/1825127452 conn(0x7fd4840fa5d0 msgr2=0x7fd4840fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:09.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.747+0000 7fd489415700 1 -- 172.21.15.52:0/1825127452 shutdown_connections 2023-12-16T20:49:09.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.747+0000 7fd489415700 1 -- 172.21.15.52:0/1825127452 wait complete. 2023-12-16T20:49:09.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.747+0000 7fd489415700 1 Processor -- start 2023-12-16T20:49:09.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.748+0000 7fd489415700 1 -- start start 2023-12-16T20:49:09.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.748+0000 7fd489415700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4840fee00 0x7fd48410f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:09.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.748+0000 7fd489415700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4840fff70 0x7fd48410fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:09.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.748+0000 7fd489415700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd484110480 con 0x7fd4840fee00 2023-12-16T20:49:09.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.748+0000 7fd489415700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd484110d20 con 0x7fd4840fff70 2023-12-16T20:49:09.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.748+0000 7fd482ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4840fee00 0x7fd48410f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:09.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.748+0000 7fd482ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4840fee00 0x7fd48410f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46120/0 (socket says 172.21.15.52:46120) 2023-12-16T20:49:09.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.748+0000 7fd482ffd700 1 -- 172.21.15.52:0/3993045751 learned_addr learned my addr 172.21.15.52:0/3993045751 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:09.751 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.748+0000 7fd4827fc700 1 --2- 172.21.15.52:0/3993045751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4840fff70 0x7fd48410fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:09.751 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.748+0000 7fd482ffd700 1 -- 172.21.15.52:0/3993045751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4840fff70 msgr2=0x7fd48410fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:09.751 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.748+0000 7fd482ffd700 1 --2- 172.21.15.52:0/3993045751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4840fff70 0x7fd48410fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:09.751 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.748+0000 7fd482ffd700 1 -- 172.21.15.52:0/3993045751 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c009940 con 0x7fd4840fee00 2023-12-16T20:49:09.751 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.748+0000 7fd482ffd700 1 --2- 172.21.15.52:0/3993045751 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4840fee00 0x7fd48410f940 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd47c009880 tx=0x7fd47c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:09.751 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.748+0000 7fd47b7fe700 1 -- 172.21.15.52:0/3993045751 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd47c00e040 con 0x7fd4840fee00 2023-12-16T20:49:09.751 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.748+0000 7fd47b7fe700 1 -- 172.21.15.52:0/3993045751 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd47c0128c0 con 0x7fd4840fee00 2023-12-16T20:49:09.751 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.749+0000 7fd47b7fe700 1 -- 172.21.15.52:0/3993045751 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd47c01c7a0 con 0x7fd4840fee00 2023-12-16T20:49:09.751 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.749+0000 7fd489415700 1 -- 172.21.15.52:0/3993045751 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd484110fa0 con 0x7fd4840fee00 2023-12-16T20:49:09.751 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.749+0000 7fd489415700 1 -- 172.21.15.52:0/3993045751 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd484111490 con 0x7fd4840fee00 2023-12-16T20:49:09.752 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.750+0000 7fd489415700 1 -- 172.21.15.52:0/3993045751 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4841097f0 con 0x7fd4840fee00 2023-12-16T20:49:09.752 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.750+0000 7fd47b7fe700 1 -- 172.21.15.52:0/3993045751 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fd47c012a30 con 0x7fd4840fee00 2023-12-16T20:49:09.752 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.750+0000 7fd47b7fe700 1 --2- 172.21.15.52:0/3993045751 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd470068f60 0x7fd47006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:09.752 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.751+0000 7fd47b7fe700 1 -- 172.21.15.52:0/3993045751 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd47c088780 con 0x7fd4840fee00 2023-12-16T20:49:09.752 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.751+0000 7fd4827fc700 1 --2- 172.21.15.52:0/3993045751 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd470068f60 0x7fd47006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:09.752 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.751+0000 7fd4827fc700 1 --2- 172.21.15.52:0/3993045751 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd470068f60 0x7fd47006b410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd46c004d60 tx=0x7fd46c00b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:09.755 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.755+0000 7fd47b7fe700 1 -- 172.21.15.52:0/3993045751 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd47c0589a0 con 0x7fd4840fee00 2023-12-16T20:49:09.964 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.964+0000 7fd489415700 1 -- 172.21.15.52:0/3993045751 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4841117c0 con 0x7fd470068f60 2023-12-16T20:49:09.965 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.965+0000 7fd47b7fe700 1 -- 172.21.15.52:0/3993045751 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fd4841117c0 con 0x7fd470068f60 2023-12-16T20:49:09.966 INFO:teuthology.orchestra.run.smithi052.stdout:Backend: cephadm 2023-12-16T20:49:09.966 INFO:teuthology.orchestra.run.smithi052.stdout:Available: Yes 2023-12-16T20:49:09.966 INFO:teuthology.orchestra.run.smithi052.stdout:Paused: No 2023-12-16T20:49:09.969 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.968+0000 7fd489415700 1 -- 172.21.15.52:0/3993045751 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd470068f60 msgr2=0x7fd47006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:09.969 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.968+0000 7fd489415700 1 --2- 172.21.15.52:0/3993045751 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd470068f60 0x7fd47006b410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd46c004d60 tx=0x7fd46c00b040 comp rx=0 tx=0).stop 2023-12-16T20:49:09.969 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.968+0000 7fd489415700 1 -- 172.21.15.52:0/3993045751 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4840fee00 msgr2=0x7fd48410f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:09.969 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.968+0000 7fd489415700 1 --2- 172.21.15.52:0/3993045751 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4840fee00 0x7fd48410f940 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd47c009880 tx=0x7fd47c004d10 comp rx=0 tx=0).stop 2023-12-16T20:49:09.969 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.968+0000 7fd489415700 1 -- 172.21.15.52:0/3993045751 shutdown_connections 2023-12-16T20:49:09.969 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.968+0000 7fd489415700 1 --2- 172.21.15.52:0/3993045751 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd470068f60 0x7fd47006b410 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:09.969 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.968+0000 7fd489415700 1 --2- 172.21.15.52:0/3993045751 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4840fee00 0x7fd48410f940 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:09.969 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.968+0000 7fd489415700 1 --2- 172.21.15.52:0/3993045751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4840fff70 0x7fd48410fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:09.969 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.968+0000 7fd489415700 1 -- 172.21.15.52:0/3993045751 >> 172.21.15.52:0/3993045751 conn(0x7fd4840fa5d0 msgr2=0x7fd4840fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:09.970 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.968+0000 7fd489415700 1 -- 172.21.15.52:0/3993045751 shutdown_connections 2023-12-16T20:49:09.970 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:09.968+0000 7fd489415700 1 -- 172.21.15.52:0/3993045751 wait complete. 2023-12-16T20:49:10.305 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- bash -c 'ceph orch ps' 2023-12-16T20:49:10.569 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:11.237 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:10 smithi052 ceph-mon[80761]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:11.237 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:10 smithi052 ceph-mon[80761]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:49:11.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:10 smithi150 ceph-mon[88000]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:11.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:10 smithi150 ceph-mon[88000]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:49:11.439 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.438+0000 7f1f910dd700 1 -- 172.21.15.52:0/3297223175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f8c0fcbf0 msgr2=0x7f1f8c0fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:11.439 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.438+0000 7f1f910dd700 1 --2- 172.21.15.52:0/3297223175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f8c0fcbf0 0x7f1f8c0fd000 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1f74004940 tx=0x7f1f74009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:11.439 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.438+0000 7f1f910dd700 1 -- 172.21.15.52:0/3297223175 shutdown_connections 2023-12-16T20:49:11.440 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.438+0000 7f1f910dd700 1 --2- 172.21.15.52:0/3297223175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f8c0fde00 0x7f1f8c0fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:11.440 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.438+0000 7f1f910dd700 1 --2- 172.21.15.52:0/3297223175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f8c0fcbf0 0x7f1f8c0fd000 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:11.440 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.438+0000 7f1f910dd700 1 -- 172.21.15.52:0/3297223175 >> 172.21.15.52:0/3297223175 conn(0x7f1f8c0f82d0 msgr2=0x7f1f8c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:11.440 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.438+0000 7f1f910dd700 1 -- 172.21.15.52:0/3297223175 shutdown_connections 2023-12-16T20:49:11.440 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.438+0000 7f1f910dd700 1 -- 172.21.15.52:0/3297223175 wait complete. 2023-12-16T20:49:11.442 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.438+0000 7f1f910dd700 1 Processor -- start 2023-12-16T20:49:11.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.438+0000 7f1f910dd700 1 -- start start 2023-12-16T20:49:11.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f910dd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f8c0fcbf0 0x7f1f8c10d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:11.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f910dd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f8c0fde00 0x7f1f8c10da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:11.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f910dd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c10e010 con 0x7f1f8c0fde00 2023-12-16T20:49:11.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f910dd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c10e8b0 con 0x7f1f8c0fcbf0 2023-12-16T20:49:11.443 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f8a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f8c0fde00 0x7f1f8c10da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:11.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f8a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f8c0fde00 0x7f1f8c10da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46148/0 (socket says 172.21.15.52:46148) 2023-12-16T20:49:11.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f8a59c700 1 -- 172.21.15.52:0/2588937944 learned_addr learned my addr 172.21.15.52:0/2588937944 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:11.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f8ad9d700 1 --2- 172.21.15.52:0/2588937944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f8c0fcbf0 0x7f1f8c10d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:11.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f8a59c700 1 -- 172.21.15.52:0/2588937944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f8c0fcbf0 msgr2=0x7f1f8c10d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:11.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f8a59c700 1 --2- 172.21.15.52:0/2588937944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f8c0fcbf0 0x7f1f8c10d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:11.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f8a59c700 1 -- 172.21.15.52:0/2588937944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f74005040 con 0x7f1f8c0fde00 2023-12-16T20:49:11.444 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f8ad9d700 1 --2- 172.21.15.52:0/2588937944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f8c0fcbf0 0x7f1f8c10d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T20:49:11.445 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f8a59c700 1 --2- 172.21.15.52:0/2588937944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f8c0fde00 0x7f1f8c10da10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00eb40 tx=0x7f1f7c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:11.445 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f837fe700 1 -- 172.21.15.52:0/2588937944 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f7c007cc0 con 0x7f1f8c0fde00 2023-12-16T20:49:11.445 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f837fe700 1 -- 172.21.15.52:0/2588937944 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f7c00f040 con 0x7f1f8c0fde00 2023-12-16T20:49:11.445 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.439+0000 7f1f837fe700 1 -- 172.21.15.52:0/2588937944 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f7c009570 con 0x7f1f8c0fde00 2023-12-16T20:49:11.445 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.440+0000 7f1f910dd700 1 -- 172.21.15.52:0/2588937944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f8c10eb30 con 0x7f1f8c0fde00 2023-12-16T20:49:11.445 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.440+0000 7f1f910dd700 1 -- 172.21.15.52:0/2588937944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8c10f020 con 0x7f1f8c0fde00 2023-12-16T20:49:11.445 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.440+0000 7f1f910dd700 1 -- 172.21.15.52:0/2588937944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8c04bb80 con 0x7f1f8c0fde00 2023-12-16T20:49:11.445 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.441+0000 7f1f837fe700 1 -- 172.21.15.52:0/2588937944 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1f7c0096d0 con 0x7f1f8c0fde00 2023-12-16T20:49:11.446 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.442+0000 7f1f837fe700 1 --2- 172.21.15.52:0/2588937944 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1f78068f60 0x7f1f7806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:11.446 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.442+0000 7f1f837fe700 1 -- 172.21.15.52:0/2588937944 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1f7c0866f0 con 0x7f1f8c0fde00 2023-12-16T20:49:11.446 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.442+0000 7f1f8ad9d700 1 --2- 172.21.15.52:0/2588937944 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1f78068f60 0x7f1f7806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:11.446 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.442+0000 7f1f8ad9d700 1 --2- 172.21.15.52:0/2588937944 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1f78068f60 0x7f1f7806b410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1f74000c00 tx=0x7f1f74009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:11.447 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.446+0000 7f1f837fe700 1 -- 172.21.15.52:0/2588937944 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1f7c04da00 con 0x7f1f8c0fde00 2023-12-16T20:49:11.655 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.653+0000 7f1f910dd700 1 -- 172.21.15.52:0/2588937944 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f8c10f4c0 con 0x7f1f78068f60 2023-12-16T20:49:11.663 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.662+0000 7f1f837fe700 1 -- 172.21.15.52:0/2588937944 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2718 (secure 0 0 0) 0x7f1f8c10f4c0 con 0x7f1f78068f60 2023-12-16T20:49:11.663 INFO:teuthology.orchestra.run.smithi052.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T20:49:11.664 INFO:teuthology.orchestra.run.smithi052.stdout:alertmanager.smithi052 smithi052 *:9093,9094 running (2m) 82s ago 3m 25.6M - 0.25.0 c8568f914cd2 70fee28ba742 2023-12-16T20:49:11.664 INFO:teuthology.orchestra.run.smithi052.stdout:crash.smithi052 smithi052 running (3m) 82s ago 3m 6665k - 17.2.7-125-g1f2f5510 60cc83a9c8f1 2cbb0d9d4ed2 2023-12-16T20:49:11.664 INFO:teuthology.orchestra.run.smithi052.stdout:crash.smithi150 smithi150 running (2m) 43s ago 2m 6665k - 17.2.7-125-g1f2f5510 60cc83a9c8f1 bb44b459c1ab 2023-12-16T20:49:11.664 INFO:teuthology.orchestra.run.smithi052.stdout:grafana.smithi052 smithi052 *:3000 running (2m) 82s ago 3m 80.4M - 9.4.7 954c08fa6188 7a62e1cb6f33 2023-12-16T20:49:11.664 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.smithi052.nmkcbs smithi052 *:9283 running (4m) 82s ago 4m 470M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 aedf615cc9aa 2023-12-16T20:49:11.664 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.smithi150.vjiupe smithi150 *:8443,9283 running (2m) 43s ago 2m 433M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 dcdbfd832a9b 2023-12-16T20:49:11.664 INFO:teuthology.orchestra.run.smithi052.stdout:mon.smithi052 smithi052 running (4m) 82s ago 4m 41.1M 2048M 17.2.7-125-g1f2f5510 60cc83a9c8f1 094cffecdaf4 2023-12-16T20:49:11.664 INFO:teuthology.orchestra.run.smithi052.stdout:mon.smithi150 smithi150 running (2m) 43s ago 2m 38.9M 2048M 17.2.7-125-g1f2f5510 60cc83a9c8f1 4e8a89294822 2023-12-16T20:49:11.664 INFO:teuthology.orchestra.run.smithi052.stdout:node-exporter.smithi052 smithi052 *:9100 running (3m) 82s ago 3m 17.1M - 1.5.0 0da6a335fe13 71e781c7e5ac 2023-12-16T20:49:11.664 INFO:teuthology.orchestra.run.smithi052.stdout:node-exporter.smithi150 smithi150 *:9100 running (2m) 43s ago 2m 16.7M - 1.5.0 0da6a335fe13 76e3676984fa 2023-12-16T20:49:11.664 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0 smithi052 running (2m) 82s ago 2m 39.1M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 70d69be336b3 2023-12-16T20:49:11.664 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1 smithi052 running (111s) 82s ago 111s 36.0M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 a39f1afa7295 2023-12-16T20:49:11.665 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2 smithi052 running (92s) 82s ago 92s 38.2M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 a382edce4272 2023-12-16T20:49:11.665 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3 smithi150 running (76s) 43s ago 76s 41.7M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 634f54a96481 2023-12-16T20:49:11.665 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4 smithi150 running (61s) 43s ago 61s 36.8M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 60f7c481a05d 2023-12-16T20:49:11.665 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5 smithi150 running (45s) 43s ago 45s 11.0M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 12c8aba080a8 2023-12-16T20:49:11.665 INFO:teuthology.orchestra.run.smithi052.stdout:prometheus.smithi052 smithi052 *:9095 running (2m) 82s ago 3m 35.8M - 2.43.0 a07b618ecd1d eebcaf89c62f 2023-12-16T20:49:11.666 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.664+0000 7f1f910dd700 1 -- 172.21.15.52:0/2588937944 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1f78068f60 msgr2=0x7f1f7806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:11.666 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.664+0000 7f1f910dd700 1 --2- 172.21.15.52:0/2588937944 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1f78068f60 0x7f1f7806b410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1f74000c00 tx=0x7f1f74009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:11.666 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.664+0000 7f1f910dd700 1 -- 172.21.15.52:0/2588937944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f8c0fde00 msgr2=0x7f1f8c10da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:11.666 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.664+0000 7f1f910dd700 1 --2- 172.21.15.52:0/2588937944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f8c0fde00 0x7f1f8c10da10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00eb40 tx=0x7f1f7c00b040 comp rx=0 tx=0).stop 2023-12-16T20:49:11.666 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.664+0000 7f1f910dd700 1 -- 172.21.15.52:0/2588937944 shutdown_connections 2023-12-16T20:49:11.666 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.664+0000 7f1f910dd700 1 --2- 172.21.15.52:0/2588937944 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1f78068f60 0x7f1f7806b410 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:11.666 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.664+0000 7f1f910dd700 1 --2- 172.21.15.52:0/2588937944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f8c0fcbf0 0x7f1f8c10d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:11.666 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.664+0000 7f1f910dd700 1 --2- 172.21.15.52:0/2588937944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f8c0fde00 0x7f1f8c10da10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:11.666 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.664+0000 7f1f910dd700 1 -- 172.21.15.52:0/2588937944 >> 172.21.15.52:0/2588937944 conn(0x7f1f8c0f82d0 msgr2=0x7f1f8c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:11.667 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.664+0000 7f1f910dd700 1 -- 172.21.15.52:0/2588937944 shutdown_connections 2023-12-16T20:49:11.667 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:11.664+0000 7f1f910dd700 1 -- 172.21.15.52:0/2588937944 wait complete. 2023-12-16T20:49:12.055 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- bash -c 'ceph orch ls' 2023-12-16T20:49:12.321 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:13.063 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:12 smithi052 ceph-mon[80761]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:49:13.063 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:12 smithi052 ceph-mon[80761]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:13.122 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.120+0000 7f4f158d1700 1 -- 172.21.15.52:0/879035811 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f100fbaf0 msgr2=0x7f4f100fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:13.123 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.120+0000 7f4f158d1700 1 --2- 172.21.15.52:0/879035811 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f100fbaf0 0x7f4f100fbf00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4f08009ce0 tx=0x7f4f08004d10 comp rx=0 tx=0).stop 2023-12-16T20:49:13.123 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.121+0000 7f4f158d1700 1 -- 172.21.15.52:0/879035811 shutdown_connections 2023-12-16T20:49:13.123 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.121+0000 7f4f158d1700 1 --2- 172.21.15.52:0/879035811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f100fcd00 0x7f4f100fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:13.123 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.121+0000 7f4f158d1700 1 --2- 172.21.15.52:0/879035811 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f100fbaf0 0x7f4f100fbf00 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:13.123 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.121+0000 7f4f158d1700 1 -- 172.21.15.52:0/879035811 >> 172.21.15.52:0/879035811 conn(0x7f4f100f7190 msgr2=0x7f4f100f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:13.123 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.121+0000 7f4f158d1700 1 -- 172.21.15.52:0/879035811 shutdown_connections 2023-12-16T20:49:13.123 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.121+0000 7f4f158d1700 1 -- 172.21.15.52:0/879035811 wait complete. 2023-12-16T20:49:13.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.121+0000 7f4f158d1700 1 Processor -- start 2023-12-16T20:49:13.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.121+0000 7f4f158d1700 1 -- start start 2023-12-16T20:49:13.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.121+0000 7f4f158d1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f100fbaf0 0x7f4f100ffbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:13.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.121+0000 7f4f158d1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f100fcd00 0x7f4f100fe040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:13.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.122+0000 7f4f158d1700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f100fe640 con 0x7f4f100fcd00 2023-12-16T20:49:13.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.122+0000 7f4f158d1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f100fe7b0 con 0x7f4f100fbaf0 2023-12-16T20:49:13.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.122+0000 7f4f0e7fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f100fcd00 0x7f4f100fe040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:13.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.122+0000 7f4f0e7fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f100fcd00 0x7f4f100fe040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43284/0 (socket says 172.21.15.52:43284) 2023-12-16T20:49:13.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.122+0000 7f4f0e7fc700 1 -- 172.21.15.52:0/3608667369 learned_addr learned my addr 172.21.15.52:0/3608667369 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:13.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.122+0000 7f4f0effd700 1 --2- 172.21.15.52:0/3608667369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f100fbaf0 0x7f4f100ffbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:13.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.122+0000 7f4f0e7fc700 1 -- 172.21.15.52:0/3608667369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f100fbaf0 msgr2=0x7f4f100ffbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:13.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.122+0000 7f4f0e7fc700 1 --2- 172.21.15.52:0/3608667369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f100fbaf0 0x7f4f100ffbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:13.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.122+0000 7f4f0e7fc700 1 -- 172.21.15.52:0/3608667369 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f08009940 con 0x7f4f100fcd00 2023-12-16T20:49:13.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.122+0000 7f4f0e7fc700 1 --2- 172.21.15.52:0/3608667369 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f100fcd00 0x7f4f100fe040 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800c8c0 tx=0x7f4ef80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:13.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.122+0000 7f4f077fe700 1 -- 172.21.15.52:0/3608667369 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ef800a980 con 0x7f4f100fcd00 2023-12-16T20:49:13.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.122+0000 7f4f158d1700 1 -- 172.21.15.52:0/3608667369 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f100fea90 con 0x7f4f100fcd00 2023-12-16T20:49:13.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.122+0000 7f4f077fe700 1 -- 172.21.15.52:0/3608667369 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ef800ce60 con 0x7f4f100fcd00 2023-12-16T20:49:13.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.122+0000 7f4f077fe700 1 -- 172.21.15.52:0/3608667369 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ef80113a0 con 0x7f4f100fcd00 2023-12-16T20:49:13.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.122+0000 7f4f158d1700 1 -- 172.21.15.52:0/3608667369 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f10115720 con 0x7f4f100fcd00 2023-12-16T20:49:13.127 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.123+0000 7f4f158d1700 1 -- 172.21.15.52:0/3608667369 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1004bb80 con 0x7f4f100fcd00 2023-12-16T20:49:13.127 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.124+0000 7f4f077fe700 1 -- 172.21.15.52:0/3608667369 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f4ef8012400 con 0x7f4f100fcd00 2023-12-16T20:49:13.127 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.124+0000 7f4f077fe700 1 --2- 172.21.15.52:0/3608667369 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4efc068f60 0x7f4efc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:13.127 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.124+0000 7f4f077fe700 1 -- 172.21.15.52:0/3608667369 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4ef8086b70 con 0x7f4f100fcd00 2023-12-16T20:49:13.127 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.124+0000 7f4f0effd700 1 --2- 172.21.15.52:0/3608667369 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4efc068f60 0x7f4efc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:13.127 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.125+0000 7f4f0effd700 1 --2- 172.21.15.52:0/3608667369 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4efc068f60 0x7f4efc06b410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4f08015040 tx=0x7f4f08004d10 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:13.130 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.129+0000 7f4f077fe700 1 -- 172.21.15.52:0/3608667369 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4ef8056e10 con 0x7f4f100fcd00 2023-12-16T20:49:13.333 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.331+0000 7f4f158d1700 1 -- 172.21.15.52:0/3608667369 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f10065a80 con 0x7f4efc068f60 2023-12-16T20:49:13.338 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.336+0000 7f4f077fe700 1 -- 172.21.15.52:0/3608667369 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1170 (secure 0 0 0) 0x7f4f10065a80 con 0x7f4efc068f60 2023-12-16T20:49:13.338 INFO:teuthology.orchestra.run.smithi052.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-12-16T20:49:13.338 INFO:teuthology.orchestra.run.smithi052.stdout:alertmanager ?:9093,9094 1/1 84s ago 3m count:1 2023-12-16T20:49:13.339 INFO:teuthology.orchestra.run.smithi052.stdout:crash 2/2 84s ago 3m * 2023-12-16T20:49:13.339 INFO:teuthology.orchestra.run.smithi052.stdout:grafana ?:3000 1/1 84s ago 3m count:1 2023-12-16T20:49:13.339 INFO:teuthology.orchestra.run.smithi052.stdout:mgr 2/2 84s ago 3m count:2 2023-12-16T20:49:13.339 INFO:teuthology.orchestra.run.smithi052.stdout:mon 2/2 84s ago 3m smithi052:172.21.15.52=smithi052;smithi150:172.21.15.150=smithi150;count:2 2023-12-16T20:49:13.339 INFO:teuthology.orchestra.run.smithi052.stdout:node-exporter ?:9100 2/2 84s ago 3m * 2023-12-16T20:49:13.339 INFO:teuthology.orchestra.run.smithi052.stdout:osd 6 84s ago - 2023-12-16T20:49:13.339 INFO:teuthology.orchestra.run.smithi052.stdout:prometheus ?:9095 1/1 84s ago 3m count:1 2023-12-16T20:49:13.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.338+0000 7f4f158d1700 1 -- 172.21.15.52:0/3608667369 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4efc068f60 msgr2=0x7f4efc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:13.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.339+0000 7f4f158d1700 1 --2- 172.21.15.52:0/3608667369 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4efc068f60 0x7f4efc06b410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4f08015040 tx=0x7f4f08004d10 comp rx=0 tx=0).stop 2023-12-16T20:49:13.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.339+0000 7f4f158d1700 1 -- 172.21.15.52:0/3608667369 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f100fcd00 msgr2=0x7f4f100fe040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:13.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.339+0000 7f4f158d1700 1 --2- 172.21.15.52:0/3608667369 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f100fcd00 0x7f4f100fe040 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800c8c0 tx=0x7f4ef80044c0 comp rx=0 tx=0).stop 2023-12-16T20:49:13.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.339+0000 7f4f158d1700 1 -- 172.21.15.52:0/3608667369 shutdown_connections 2023-12-16T20:49:13.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.339+0000 7f4f158d1700 1 --2- 172.21.15.52:0/3608667369 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4efc068f60 0x7f4efc06b410 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:13.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.339+0000 7f4f158d1700 1 --2- 172.21.15.52:0/3608667369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f100fbaf0 0x7f4f100ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:13.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.339+0000 7f4f158d1700 1 --2- 172.21.15.52:0/3608667369 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f100fcd00 0x7f4f100fe040 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:13.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.339+0000 7f4f158d1700 1 -- 172.21.15.52:0/3608667369 >> 172.21.15.52:0/3608667369 conn(0x7f4f100f7190 msgr2=0x7f4f10065370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:13.341 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.339+0000 7f4f158d1700 1 -- 172.21.15.52:0/3608667369 shutdown_connections 2023-12-16T20:49:13.342 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:13.339+0000 7f4f158d1700 1 -- 172.21.15.52:0/3608667369 wait complete. 2023-12-16T20:49:13.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:12 smithi150 ceph-mon[88000]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:49:13.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:12 smithi150 ceph-mon[88000]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:13.673 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- bash -c 'ceph orch host ls' 2023-12-16T20:49:13.937 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:14.808 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.805+0000 7f2f4789e700 1 -- 172.21.15.52:0/2085790137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f400ffd90 msgr2=0x7f2f401021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:14.808 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.805+0000 7f2f4789e700 1 --2- 172.21.15.52:0/2085790137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f400ffd90 0x7f2f401021d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2f34004940 tx=0x7f2f34009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:14.808 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.805+0000 7f2f4789e700 1 -- 172.21.15.52:0/2085790137 shutdown_connections 2023-12-16T20:49:14.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.805+0000 7f2f4789e700 1 --2- 172.21.15.52:0/2085790137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f400ffd90 0x7f2f401021d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:14.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.805+0000 7f2f4789e700 1 --2- 172.21.15.52:0/2085790137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f40066940 0x7f2f400ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:14.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.805+0000 7f2f4789e700 1 -- 172.21.15.52:0/2085790137 >> 172.21.15.52:0/2085790137 conn(0x7f2f400f71d0 msgr2=0x7f2f400f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:14.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.805+0000 7f2f4789e700 1 -- 172.21.15.52:0/2085790137 shutdown_connections 2023-12-16T20:49:14.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.805+0000 7f2f4789e700 1 -- 172.21.15.52:0/2085790137 wait complete. 2023-12-16T20:49:14.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.806+0000 7f2f4789e700 1 Processor -- start 2023-12-16T20:49:14.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.806+0000 7f2f4789e700 1 -- start start 2023-12-16T20:49:14.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.806+0000 7f2f4789e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f40066940 0x7f2f4010d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:14.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.806+0000 7f2f4789e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f400ffd90 0x7f2f4010da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:14.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.806+0000 7f2f4789e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4010e0f0 con 0x7f2f40066940 2023-12-16T20:49:14.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.806+0000 7f2f4789e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4010e930 con 0x7f2f400ffd90 2023-12-16T20:49:14.810 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.806+0000 7f2f4563a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f40066940 0x7f2f4010d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:14.810 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.806+0000 7f2f4563a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f40066940 0x7f2f4010d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43300/0 (socket says 172.21.15.52:43300) 2023-12-16T20:49:14.810 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.806+0000 7f2f4563a700 1 -- 172.21.15.52:0/2025193638 learned_addr learned my addr 172.21.15.52:0/2025193638 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:14.810 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.807+0000 7f2f44e39700 1 --2- 172.21.15.52:0/2025193638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f400ffd90 0x7f2f4010da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:14.810 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.807+0000 7f2f4563a700 1 -- 172.21.15.52:0/2025193638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f400ffd90 msgr2=0x7f2f4010da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:14.810 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.807+0000 7f2f4563a700 1 --2- 172.21.15.52:0/2025193638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f400ffd90 0x7f2f4010da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:14.810 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.807+0000 7f2f4563a700 1 -- 172.21.15.52:0/2025193638 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f34005040 con 0x7f2f40066940 2023-12-16T20:49:14.811 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.807+0000 7f2f4563a700 1 --2- 172.21.15.52:0/2025193638 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f40066940 0x7f2f4010d520 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800bbb0 tx=0x7f2f38009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:14.811 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.807+0000 7f2f3dffb700 1 -- 172.21.15.52:0/2025193638 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f380092e0 con 0x7f2f40066940 2023-12-16T20:49:14.811 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.807+0000 7f2f3dffb700 1 -- 172.21.15.52:0/2025193638 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f3800cbb0 con 0x7f2f40066940 2023-12-16T20:49:14.811 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.807+0000 7f2f4789e700 1 -- 172.21.15.52:0/2025193638 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f4010ec10 con 0x7f2f40066940 2023-12-16T20:49:14.812 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.807+0000 7f2f3dffb700 1 -- 172.21.15.52:0/2025193638 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f38015d70 con 0x7f2f40066940 2023-12-16T20:49:14.812 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.808+0000 7f2f4789e700 1 -- 172.21.15.52:0/2025193638 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f4010f150 con 0x7f2f40066940 2023-12-16T20:49:14.812 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.808+0000 7f2f4789e700 1 -- 172.21.15.52:0/2025193638 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f40107640 con 0x7f2f40066940 2023-12-16T20:49:14.812 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.809+0000 7f2f3dffb700 1 -- 172.21.15.52:0/2025193638 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f2f38012070 con 0x7f2f40066940 2023-12-16T20:49:14.812 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.809+0000 7f2f3dffb700 1 --2- 172.21.15.52:0/2025193638 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2f28068f70 0x7f2f2806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:14.812 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.809+0000 7f2f3dffb700 1 -- 172.21.15.52:0/2025193638 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2f380865e0 con 0x7f2f40066940 2023-12-16T20:49:14.812 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.809+0000 7f2f44e39700 1 --2- 172.21.15.52:0/2025193638 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2f28068f70 0x7f2f2806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:14.812 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.810+0000 7f2f44e39700 1 --2- 172.21.15.52:0/2025193638 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2f28068f70 0x7f2f2806b420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400b3b0 tx=0x7f2f34009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:14.815 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:14.814+0000 7f2f3dffb700 1 -- 172.21.15.52:0/2025193638 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2f38056880 con 0x7f2f40066940 2023-12-16T20:49:15.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:14 smithi052 ceph-mon[80761]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:49:15.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:14 smithi052 ceph-mon[80761]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:15.016 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:15.015+0000 7f2f4789e700 1 -- 172.21.15.52:0/2025193638 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f4010f430 con 0x7f2f28068f70 2023-12-16T20:49:15.018 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:15.016+0000 7f2f3dffb700 1 -- 172.21.15.52:0/2025193638 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f2f4010f430 con 0x7f2f28068f70 2023-12-16T20:49:15.018 INFO:teuthology.orchestra.run.smithi052.stdout:HOST ADDR LABELS STATUS 2023-12-16T20:49:15.018 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 172.21.15.52 2023-12-16T20:49:15.018 INFO:teuthology.orchestra.run.smithi052.stdout:smithi150 172.21.15.150 2023-12-16T20:49:15.018 INFO:teuthology.orchestra.run.smithi052.stdout:2 hosts in cluster 2023-12-16T20:49:15.020 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:15.018+0000 7f2f4789e700 1 -- 172.21.15.52:0/2025193638 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2f28068f70 msgr2=0x7f2f2806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:15.020 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:15.018+0000 7f2f4789e700 1 --2- 172.21.15.52:0/2025193638 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2f28068f70 0x7f2f2806b420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400b3b0 tx=0x7f2f34009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:15.020 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:15.018+0000 7f2f4789e700 1 -- 172.21.15.52:0/2025193638 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f40066940 msgr2=0x7f2f4010d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:15.020 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:15.018+0000 7f2f4789e700 1 --2- 172.21.15.52:0/2025193638 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f40066940 0x7f2f4010d520 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800bbb0 tx=0x7f2f38009840 comp rx=0 tx=0).stop 2023-12-16T20:49:15.020 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:15.018+0000 7f2f4789e700 1 -- 172.21.15.52:0/2025193638 shutdown_connections 2023-12-16T20:49:15.020 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:15.018+0000 7f2f4789e700 1 --2- 172.21.15.52:0/2025193638 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2f28068f70 0x7f2f2806b420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:15.021 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:15.018+0000 7f2f4789e700 1 --2- 172.21.15.52:0/2025193638 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f40066940 0x7f2f4010d520 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:15.021 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:15.018+0000 7f2f4789e700 1 --2- 172.21.15.52:0/2025193638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f400ffd90 0x7f2f4010da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:15.021 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:15.018+0000 7f2f4789e700 1 -- 172.21.15.52:0/2025193638 >> 172.21.15.52:0/2025193638 conn(0x7f2f400f71d0 msgr2=0x7f2f400f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:15.021 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:15.018+0000 7f2f4789e700 1 -- 172.21.15.52:0/2025193638 shutdown_connections 2023-12-16T20:49:15.021 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:15.018+0000 7f2f4789e700 1 -- 172.21.15.52:0/2025193638 wait complete. 2023-12-16T20:49:15.389 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- bash -c 'ceph orch device ls' 2023-12-16T20:49:15.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:14 smithi150 ceph-mon[88000]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:49:15.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:14 smithi150 ceph-mon[88000]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:15.655 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:16.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:15 smithi052 ceph-mon[80761]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:49:16.253 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:15 smithi150 ceph-mon[88000]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:49:16.558 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.555+0000 7fe42f355700 1 -- 172.21.15.52:0/1400816597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4280fcbf0 msgr2=0x7fe4280fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:16.559 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.555+0000 7fe42f355700 1 --2- 172.21.15.52:0/1400816597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4280fcbf0 0x7fe4280fd000 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe418004940 tx=0x7fe418009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:16.559 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.555+0000 7fe42f355700 1 -- 172.21.15.52:0/1400816597 shutdown_connections 2023-12-16T20:49:16.559 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.555+0000 7fe42f355700 1 --2- 172.21.15.52:0/1400816597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4280fde00 0x7fe4280fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:16.559 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.555+0000 7fe42f355700 1 --2- 172.21.15.52:0/1400816597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4280fcbf0 0x7fe4280fd000 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:16.559 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.555+0000 7fe42f355700 1 -- 172.21.15.52:0/1400816597 >> 172.21.15.52:0/1400816597 conn(0x7fe4280f82d0 msgr2=0x7fe4280fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:16.559 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.555+0000 7fe42f355700 1 -- 172.21.15.52:0/1400816597 shutdown_connections 2023-12-16T20:49:16.559 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.555+0000 7fe42f355700 1 -- 172.21.15.52:0/1400816597 wait complete. 2023-12-16T20:49:16.559 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.556+0000 7fe42f355700 1 Processor -- start 2023-12-16T20:49:16.559 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.556+0000 7fe42f355700 1 -- start start 2023-12-16T20:49:16.559 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.556+0000 7fe42f355700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4280fcbf0 0x7fe42810d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:16.559 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.556+0000 7fe42f355700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4280fde00 0x7fe42810dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:16.559 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.556+0000 7fe42f355700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42810e140 con 0x7fe4280fcbf0 2023-12-16T20:49:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.556+0000 7fe42f355700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42810e980 con 0x7fe4280fde00 2023-12-16T20:49:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.556+0000 7fe42d0f1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4280fcbf0 0x7fe42810d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.556+0000 7fe42d0f1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4280fcbf0 0x7fe42810d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43316/0 (socket says 172.21.15.52:43316) 2023-12-16T20:49:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.556+0000 7fe42d0f1700 1 -- 172.21.15.52:0/1767588986 learned_addr learned my addr 172.21.15.52:0/1767588986 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.556+0000 7fe42c8f0700 1 --2- 172.21.15.52:0/1767588986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4280fde00 0x7fe42810dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.557+0000 7fe42d0f1700 1 -- 172.21.15.52:0/1767588986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4280fde00 msgr2=0x7fe42810dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.557+0000 7fe42d0f1700 1 --2- 172.21.15.52:0/1767588986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4280fde00 0x7fe42810dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.557+0000 7fe42d0f1700 1 -- 172.21.15.52:0/1767588986 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe418005040 con 0x7fe4280fcbf0 2023-12-16T20:49:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.557+0000 7fe42d0f1700 1 --2- 172.21.15.52:0/1767588986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4280fcbf0 0x7fe42810d570 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe418004f90 tx=0x7fe418009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.557+0000 7fe41dffb700 1 -- 172.21.15.52:0/1767588986 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe41800e040 con 0x7fe4280fcbf0 2023-12-16T20:49:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.557+0000 7fe41dffb700 1 -- 172.21.15.52:0/1767588986 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe41800bcd0 con 0x7fe4280fcbf0 2023-12-16T20:49:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.557+0000 7fe42f355700 1 -- 172.21.15.52:0/1767588986 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42810ec00 con 0x7fe4280fcbf0 2023-12-16T20:49:16.561 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.557+0000 7fe41dffb700 1 -- 172.21.15.52:0/1767588986 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe41801c630 con 0x7fe4280fcbf0 2023-12-16T20:49:16.561 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.557+0000 7fe42f355700 1 -- 172.21.15.52:0/1767588986 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42810f140 con 0x7fe4280fcbf0 2023-12-16T20:49:16.562 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.558+0000 7fe42f355700 1 -- 172.21.15.52:0/1767588986 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe42804bb80 con 0x7fe4280fcbf0 2023-12-16T20:49:16.562 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.559+0000 7fe41dffb700 1 -- 172.21.15.52:0/1767588986 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe418013420 con 0x7fe4280fcbf0 2023-12-16T20:49:16.562 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.559+0000 7fe41dffb700 1 --2- 172.21.15.52:0/1767588986 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe414068f60 0x7fe41406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:16.562 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.559+0000 7fe41dffb700 1 -- 172.21.15.52:0/1767588986 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe418087dd0 con 0x7fe4280fcbf0 2023-12-16T20:49:16.562 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.559+0000 7fe42c8f0700 1 --2- 172.21.15.52:0/1767588986 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe414068f60 0x7fe41406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:16.562 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.560+0000 7fe42c8f0700 1 --2- 172.21.15.52:0/1767588986 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe414068f60 0x7fe41406b410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe424006fd0 tx=0x7fe424008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:16.565 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.564+0000 7fe41dffb700 1 -- 172.21.15.52:0/1767588986 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe418058110 con 0x7fe4280fcbf0 2023-12-16T20:49:16.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.766+0000 7fe42f355700 1 -- 172.21.15.52:0/1767588986 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe42810f650 con 0x7fe414068f60 2023-12-16T20:49:16.769 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.768+0000 7fe41dffb700 1 -- 172.21.15.52:0/1767588986 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7fe42810f650 con 0x7fe414068f60 2023-12-16T20:49:16.770 INFO:teuthology.orchestra.run.smithi052.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-12-16T20:49:16.770 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513010N480DGN 447G No 90s ago Has a FileSystem, LVM detected 2023-12-16T20:49:16.770 INFO:teuthology.orchestra.run.smithi052.stdout:smithi150 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GX400BGN 372G No 46s ago Has a FileSystem, LVM detected 2023-12-16T20:49:16.772 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.770+0000 7fe42f355700 1 -- 172.21.15.52:0/1767588986 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe414068f60 msgr2=0x7fe41406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:16.772 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.770+0000 7fe42f355700 1 --2- 172.21.15.52:0/1767588986 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe414068f60 0x7fe41406b410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe424006fd0 tx=0x7fe424008040 comp rx=0 tx=0).stop 2023-12-16T20:49:16.773 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.770+0000 7fe42f355700 1 -- 172.21.15.52:0/1767588986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4280fcbf0 msgr2=0x7fe42810d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:16.773 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.770+0000 7fe42f355700 1 --2- 172.21.15.52:0/1767588986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4280fcbf0 0x7fe42810d570 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe418004f90 tx=0x7fe418009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:16.773 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.770+0000 7fe42f355700 1 -- 172.21.15.52:0/1767588986 shutdown_connections 2023-12-16T20:49:16.773 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.770+0000 7fe42f355700 1 --2- 172.21.15.52:0/1767588986 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe414068f60 0x7fe41406b410 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:16.773 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.770+0000 7fe42f355700 1 --2- 172.21.15.52:0/1767588986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4280fcbf0 0x7fe42810d570 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:16.773 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.770+0000 7fe42f355700 1 --2- 172.21.15.52:0/1767588986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4280fde00 0x7fe42810dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:16.773 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.770+0000 7fe42f355700 1 -- 172.21.15.52:0/1767588986 >> 172.21.15.52:0/1767588986 conn(0x7fe4280f82d0 msgr2=0x7fe428101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:16.773 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.770+0000 7fe42f355700 1 -- 172.21.15.52:0/1767588986 shutdown_connections 2023-12-16T20:49:16.773 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:16.770+0000 7fe42f355700 1 -- 172.21.15.52:0/1767588986 wait complete. 2023-12-16T20:49:17.129 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T20:49:17.140 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi052.front.sepia.ceph.com 2023-12-16T20:49:17.140 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2023-12-16T20:49:17.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:16 smithi052 ceph-mon[80761]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:17.407 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:17.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:16 smithi150 ceph-mon[88000]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:18.246 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:17 smithi052 ceph-mon[80761]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:49:18.246 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:17 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:49:18.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.348+0000 7f2ab066c700 1 -- 172.21.15.52:0/554964747 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2aa80fbe30 msgr2=0x7f2aa80fc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:18.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.348+0000 7f2ab066c700 1 --2- 172.21.15.52:0/554964747 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2aa80fbe30 0x7f2aa80fc2a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2a98004940 tx=0x7f2a98009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:18.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.348+0000 7f2ab066c700 1 -- 172.21.15.52:0/554964747 shutdown_connections 2023-12-16T20:49:18.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.348+0000 7f2ab066c700 1 --2- 172.21.15.52:0/554964747 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2aa80fbe30 0x7f2aa80fc2a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:18.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.348+0000 7f2ab066c700 1 --2- 172.21.15.52:0/554964747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2aa80fd120 0x7f2aa80fd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:18.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.348+0000 7f2ab066c700 1 -- 172.21.15.52:0/554964747 >> 172.21.15.52:0/554964747 conn(0x7f2aa80f6950 msgr2=0x7f2aa80f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:18.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.348+0000 7f2ab066c700 1 -- 172.21.15.52:0/554964747 shutdown_connections 2023-12-16T20:49:18.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.348+0000 7f2ab066c700 1 -- 172.21.15.52:0/554964747 wait complete. 2023-12-16T20:49:18.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.349+0000 7f2ab066c700 1 Processor -- start 2023-12-16T20:49:18.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.349+0000 7f2ab066c700 1 -- start start 2023-12-16T20:49:18.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.349+0000 7f2ab066c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2aa80fbe30 0x7f2aa81102c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:18.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.349+0000 7f2ab066c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2aa80fd120 0x7f2aa8112810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:18.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.349+0000 7f2ab066c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa80fbb20 con 0x7f2aa80fbe30 2023-12-16T20:49:18.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.349+0000 7f2ab066c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa80fbc60 con 0x7f2aa80fd120 2023-12-16T20:49:18.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.349+0000 7f2aae408700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2aa80fbe30 0x7f2aa81102c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:18.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.349+0000 7f2aae408700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2aa80fbe30 0x7f2aa81102c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43326/0 (socket says 172.21.15.52:43326) 2023-12-16T20:49:18.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.349+0000 7f2aae408700 1 -- 172.21.15.52:0/1520879660 learned_addr learned my addr 172.21.15.52:0/1520879660 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:18.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.349+0000 7f2aadc07700 1 --2- 172.21.15.52:0/1520879660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2aa80fd120 0x7f2aa8112810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:18.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.350+0000 7f2aae408700 1 -- 172.21.15.52:0/1520879660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2aa80fd120 msgr2=0x7f2aa8112810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:18.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.350+0000 7f2aae408700 1 --2- 172.21.15.52:0/1520879660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2aa80fd120 0x7f2aa8112810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:18.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.350+0000 7f2aae408700 1 -- 172.21.15.52:0/1520879660 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a98005040 con 0x7f2aa80fbe30 2023-12-16T20:49:18.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.350+0000 7f2aae408700 1 --2- 172.21.15.52:0/1520879660 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2aa80fbe30 0x7f2aa81102c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400c910 tx=0x7f2aa40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:18.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.350+0000 7f2a9effd700 1 -- 172.21.15.52:0/1520879660 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aa400a790 con 0x7f2aa80fbe30 2023-12-16T20:49:18.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.350+0000 7f2ab066c700 1 -- 172.21.15.52:0/1520879660 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa8112e70 con 0x7f2aa80fbe30 2023-12-16T20:49:18.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.350+0000 7f2a9effd700 1 -- 172.21.15.52:0/1520879660 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2aa400ada0 con 0x7f2aa80fbe30 2023-12-16T20:49:18.354 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.350+0000 7f2a9effd700 1 -- 172.21.15.52:0/1520879660 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aa4012ea0 con 0x7f2aa80fbe30 2023-12-16T20:49:18.354 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.350+0000 7f2ab066c700 1 -- 172.21.15.52:0/1520879660 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa8113320 con 0x7f2aa80fbe30 2023-12-16T20:49:18.354 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.351+0000 7f2ab066c700 1 -- 172.21.15.52:0/1520879660 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa804bb80 con 0x7f2aa80fbe30 2023-12-16T20:49:18.355 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.351+0000 7f2a9effd700 1 -- 172.21.15.52:0/1520879660 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f2aa400a8f0 con 0x7f2aa80fbe30 2023-12-16T20:49:18.355 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.352+0000 7f2a9effd700 1 --2- 172.21.15.52:0/1520879660 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2a94068f60 0x7f2a9406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:18.355 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.352+0000 7f2a9effd700 1 -- 172.21.15.52:0/1520879660 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2aa4086b00 con 0x7f2aa80fbe30 2023-12-16T20:49:18.355 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.352+0000 7f2aadc07700 1 --2- 172.21.15.52:0/1520879660 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2a94068f60 0x7f2a9406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:18.355 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.352+0000 7f2aadc07700 1 --2- 172.21.15.52:0/1520879660 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2a94068f60 0x7f2a9406b410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2a98004f90 tx=0x7f2a98009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:18.358 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.357+0000 7f2a9effd700 1 -- 172.21.15.52:0/1520879660 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2aa4053310 con 0x7f2aa80fbe30 2023-12-16T20:49:18.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:17 smithi150 ceph-mon[88000]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:49:18.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:17 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:49:18.580 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:18.579+0000 7f2ab066c700 1 -- 172.21.15.52:0/1520879660 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2aa8113570 con 0x7f2a94068f60 2023-12-16T20:49:19.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:19 smithi150 ceph-mon[88000]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:19.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:19 smithi052 ceph-mon[80761]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:20.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:20 smithi150 ceph-mon[88000]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:49:20.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:20 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2023-12-16T20:49:20.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:20 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2023-12-16T20:49:20.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:20 smithi150 ceph-mon[88000]: osdmap e38: 6 total, 6 up, 6 in 2023-12-16T20:49:20.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:20 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2023-12-16T20:49:20.454 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:20 smithi052 ceph-mon[80761]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:49:20.454 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:20 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2023-12-16T20:49:20.454 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:20 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2023-12-16T20:49:20.454 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:20 smithi052 ceph-mon[80761]: osdmap e38: 6 total, 6 up, 6 in 2023-12-16T20:49:20.454 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:20 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2023-12-16T20:49:20.469 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:20.467+0000 7f2a9effd700 1 -- 172.21.15.52:0/1520879660 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2aa8113570 con 0x7f2a94068f60 2023-12-16T20:49:20.474 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:20.471+0000 7f2ab066c700 1 -- 172.21.15.52:0/1520879660 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2a94068f60 msgr2=0x7f2a9406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:20.474 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:20.471+0000 7f2ab066c700 1 --2- 172.21.15.52:0/1520879660 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2a94068f60 0x7f2a9406b410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2a98004f90 tx=0x7f2a98009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:20.474 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:20.471+0000 7f2ab066c700 1 -- 172.21.15.52:0/1520879660 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2aa80fbe30 msgr2=0x7f2aa81102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:20.474 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:20.471+0000 7f2ab066c700 1 --2- 172.21.15.52:0/1520879660 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2aa80fbe30 0x7f2aa81102c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400c910 tx=0x7f2aa40044c0 comp rx=0 tx=0).stop 2023-12-16T20:49:20.475 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:20.473+0000 7f2ab066c700 1 -- 172.21.15.52:0/1520879660 shutdown_connections 2023-12-16T20:49:20.475 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:20.473+0000 7f2ab066c700 1 --2- 172.21.15.52:0/1520879660 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2a94068f60 0x7f2a9406b410 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:20.475 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:20.473+0000 7f2ab066c700 1 --2- 172.21.15.52:0/1520879660 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2aa80fbe30 0x7f2aa81102c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:20.475 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:20.473+0000 7f2ab066c700 1 --2- 172.21.15.52:0/1520879660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2aa80fd120 0x7f2aa8112810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:20.475 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:20.473+0000 7f2ab066c700 1 -- 172.21.15.52:0/1520879660 >> 172.21.15.52:0/1520879660 conn(0x7f2aa80f6950 msgr2=0x7f2aa8100070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:20.475 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:20.474+0000 7f2ab066c700 1 -- 172.21.15.52:0/1520879660 shutdown_connections 2023-12-16T20:49:20.475 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:20.474+0000 7f2ab066c700 1 -- 172.21.15.52:0/1520879660 wait complete. 2023-12-16T20:49:20.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:49:20.448+0000 7f671a2f0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-16T20:49:20.813 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- bash -c 'ceph fs dump' 2023-12-16T20:49:21.087 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: pgmap v117: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:21.088 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-16T20:49:21.088 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2023-12-16T20:49:21.088 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: osdmap e39: 6 total, 6 up, 6 in 2023-12-16T20:49:21.088 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2023-12-16T20:49:21.088 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-16T20:49:21.088 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-12-16T20:49:21.088 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: osdmap e40: 6 total, 6 up, 6 in 2023-12-16T20:49:21.088 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2023-12-16T20:49:21.088 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: fsmap cephfs:0 2023-12-16T20:49:21.088 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:21.088 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:49:21.088 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:21.089 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:49:21.089 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:21.089 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.bvmosa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T20:49:21.089 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.bvmosa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T20:49:21.089 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:21 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:21.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: pgmap v117: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:21.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-16T20:49:21.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2023-12-16T20:49:21.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: osdmap e39: 6 total, 6 up, 6 in 2023-12-16T20:49:21.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2023-12-16T20:49:21.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-16T20:49:21.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-12-16T20:49:21.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: osdmap e40: 6 total, 6 up, 6 in 2023-12-16T20:49:21.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2023-12-16T20:49:21.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: fsmap cephfs:0 2023-12-16T20:49:21.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:21.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:49:21.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:21.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:49:21.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:21.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.bvmosa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T20:49:21.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.bvmosa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T20:49:21.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:21 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:21.446 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:22.278 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:22 smithi052 ceph-mon[80761]: Saving service mds.cephfs spec with placement count:4 2023-12-16T20:49:22.278 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:22 smithi052 ceph-mon[80761]: Deploying daemon mds.cephfs.smithi052.bvmosa on smithi052 2023-12-16T20:49:22.278 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:22 smithi052 ceph-mon[80761]: osdmap e41: 6 total, 6 up, 6 in 2023-12-16T20:49:22.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:22 smithi150 ceph-mon[88000]: Saving service mds.cephfs spec with placement count:4 2023-12-16T20:49:22.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:22 smithi150 ceph-mon[88000]: Deploying daemon mds.cephfs.smithi052.bvmosa on smithi052 2023-12-16T20:49:22.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:22 smithi150 ceph-mon[88000]: osdmap e41: 6 total, 6 up, 6 in 2023-12-16T20:49:22.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.522+0000 7fbc36037700 1 -- 172.21.15.52:0/3095819054 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc300fde00 msgr2=0x7fbc300fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:22.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.522+0000 7fbc36037700 1 --2- 172.21.15.52:0/3095819054 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc300fde00 0x7fbc300fe250 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbc20004940 tx=0x7fbc20009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:22.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.522+0000 7fbc36037700 1 -- 172.21.15.52:0/3095819054 shutdown_connections 2023-12-16T20:49:22.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.522+0000 7fbc36037700 1 --2- 172.21.15.52:0/3095819054 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc300fde00 0x7fbc300fe250 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:22.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.522+0000 7fbc36037700 1 --2- 172.21.15.52:0/3095819054 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc300fcbf0 0x7fbc300fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:22.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.522+0000 7fbc36037700 1 -- 172.21.15.52:0/3095819054 >> 172.21.15.52:0/3095819054 conn(0x7fbc300f82d0 msgr2=0x7fbc300fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:22.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.522+0000 7fbc36037700 1 -- 172.21.15.52:0/3095819054 shutdown_connections 2023-12-16T20:49:22.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.522+0000 7fbc36037700 1 -- 172.21.15.52:0/3095819054 wait complete. 2023-12-16T20:49:22.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.523+0000 7fbc36037700 1 Processor -- start 2023-12-16T20:49:22.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.523+0000 7fbc36037700 1 -- start start 2023-12-16T20:49:22.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.523+0000 7fbc36037700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc300fcbf0 0x7fbc30071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:22.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.523+0000 7fbc36037700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc300fde00 0x7fbc3006ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:22.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.523+0000 7fbc36037700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc30070590 con 0x7fbc300fcbf0 2023-12-16T20:49:22.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.523+0000 7fbc36037700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc30070700 con 0x7fbc300fde00 2023-12-16T20:49:22.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.523+0000 7fbc2f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc300fcbf0 0x7fbc30071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:22.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.524+0000 7fbc2f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc300fcbf0 0x7fbc30071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43334/0 (socket says 172.21.15.52:43334) 2023-12-16T20:49:22.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.524+0000 7fbc2f7fe700 1 -- 172.21.15.52:0/3051817235 learned_addr learned my addr 172.21.15.52:0/3051817235 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:22.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.524+0000 7fbc2effd700 1 --2- 172.21.15.52:0/3051817235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc300fde00 0x7fbc3006ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:22.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.524+0000 7fbc2f7fe700 1 -- 172.21.15.52:0/3051817235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc300fde00 msgr2=0x7fbc3006ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:22.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.524+0000 7fbc2f7fe700 1 --2- 172.21.15.52:0/3051817235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc300fde00 0x7fbc3006ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:22.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.524+0000 7fbc2f7fe700 1 -- 172.21.15.52:0/3051817235 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc20005040 con 0x7fbc300fcbf0 2023-12-16T20:49:22.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.524+0000 7fbc2effd700 1 --2- 172.21.15.52:0/3051817235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc300fde00 0x7fbc3006ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T20:49:22.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.524+0000 7fbc2f7fe700 1 --2- 172.21.15.52:0/3051817235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc300fcbf0 0x7fbc30071b10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400c8b0 tx=0x7fbc24004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:22.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.524+0000 7fbc17fff700 1 -- 172.21.15.52:0/3051817235 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc2400a760 con 0x7fbc300fcbf0 2023-12-16T20:49:22.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.524+0000 7fbc17fff700 1 -- 172.21.15.52:0/3051817235 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc2400ad70 con 0x7fbc300fcbf0 2023-12-16T20:49:22.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.524+0000 7fbc36037700 1 -- 172.21.15.52:0/3051817235 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc300709e0 con 0x7fbc300fcbf0 2023-12-16T20:49:22.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.524+0000 7fbc17fff700 1 -- 172.21.15.52:0/3051817235 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc240113a0 con 0x7fbc300fcbf0 2023-12-16T20:49:22.528 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.524+0000 7fbc36037700 1 -- 172.21.15.52:0/3051817235 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc301155c0 con 0x7fbc300fcbf0 2023-12-16T20:49:22.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.525+0000 7fbc36037700 1 -- 172.21.15.52:0/3051817235 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3004bb80 con 0x7fbc300fcbf0 2023-12-16T20:49:22.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.526+0000 7fbc17fff700 1 -- 172.21.15.52:0/3051817235 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fbc24012400 con 0x7fbc300fcbf0 2023-12-16T20:49:22.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.526+0000 7fbc17fff700 1 --2- 172.21.15.52:0/3051817235 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fbc18068f60 0x7fbc1806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:22.529 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.526+0000 7fbc17fff700 1 -- 172.21.15.52:0/3051817235 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fbc24086ce0 con 0x7fbc300fcbf0 2023-12-16T20:49:22.530 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.526+0000 7fbc2effd700 1 --2- 172.21.15.52:0/3051817235 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fbc18068f60 0x7fbc1806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:22.530 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.527+0000 7fbc2effd700 1 --2- 172.21.15.52:0/3051817235 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fbc18068f60 0x7fbc1806b410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbc20004560 tx=0x7fbc20009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:22.532 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.531+0000 7fbc17fff700 1 -- 172.21.15.52:0/3051817235 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbc24056ce0 con 0x7fbc300fcbf0 2023-12-16T20:49:22.780 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.778+0000 7fbc36037700 1 -- 172.21.15.52:0/3051817235 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbc30115a30 con 0x7fbc300fcbf0 2023-12-16T20:49:22.780 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.779+0000 7fbc17fff700 1 -- 172.21.15.52:0/3051817235 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7fbc24056870 con 0x7fbc300fcbf0 2023-12-16T20:49:22.781 INFO:teuthology.orchestra.run.smithi052.stdout:e2 2023-12-16T20:49:22.781 INFO:teuthology.orchestra.run.smithi052.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-12-16T20:49:22.781 INFO:teuthology.orchestra.run.smithi052.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T20:49:22.781 INFO:teuthology.orchestra.run.smithi052.stdout:legacy client fscid: 1 2023-12-16T20:49:22.781 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:49:22.782 INFO:teuthology.orchestra.run.smithi052.stdout:Filesystem 'cephfs' (1) 2023-12-16T20:49:22.782 INFO:teuthology.orchestra.run.smithi052.stdout:fs_name cephfs 2023-12-16T20:49:22.782 INFO:teuthology.orchestra.run.smithi052.stdout:epoch 2 2023-12-16T20:49:22.782 INFO:teuthology.orchestra.run.smithi052.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2023-12-16T20:49:22.782 INFO:teuthology.orchestra.run.smithi052.stdout:created 2023-12-16T20:49:20.450316+0000 2023-12-16T20:49:22.782 INFO:teuthology.orchestra.run.smithi052.stdout:modified 2023-12-16T20:49:20.450377+0000 2023-12-16T20:49:22.783 INFO:teuthology.orchestra.run.smithi052.stdout:tableserver 0 2023-12-16T20:49:22.783 INFO:teuthology.orchestra.run.smithi052.stdout:root 0 2023-12-16T20:49:22.784 INFO:teuthology.orchestra.run.smithi052.stdout:session_timeout 60 2023-12-16T20:49:22.784 INFO:teuthology.orchestra.run.smithi052.stdout:session_autoclose 300 2023-12-16T20:49:22.784 INFO:teuthology.orchestra.run.smithi052.stdout:max_file_size 1099511627776 2023-12-16T20:49:22.784 INFO:teuthology.orchestra.run.smithi052.stdout:required_client_features {} 2023-12-16T20:49:22.784 INFO:teuthology.orchestra.run.smithi052.stdout:last_failure 0 2023-12-16T20:49:22.784 INFO:teuthology.orchestra.run.smithi052.stdout:last_failure_osd_epoch 0 2023-12-16T20:49:22.785 INFO:teuthology.orchestra.run.smithi052.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T20:49:22.785 INFO:teuthology.orchestra.run.smithi052.stdout:max_mds 1 2023-12-16T20:49:22.785 INFO:teuthology.orchestra.run.smithi052.stdout:in 2023-12-16T20:49:22.785 INFO:teuthology.orchestra.run.smithi052.stdout:up {} 2023-12-16T20:49:22.785 INFO:teuthology.orchestra.run.smithi052.stdout:failed 2023-12-16T20:49:22.785 INFO:teuthology.orchestra.run.smithi052.stdout:damaged 2023-12-16T20:49:22.785 INFO:teuthology.orchestra.run.smithi052.stdout:stopped 2023-12-16T20:49:22.785 INFO:teuthology.orchestra.run.smithi052.stdout:data_pools [3] 2023-12-16T20:49:22.785 INFO:teuthology.orchestra.run.smithi052.stdout:metadata_pool 2 2023-12-16T20:49:22.785 INFO:teuthology.orchestra.run.smithi052.stdout:inline_data disabled 2023-12-16T20:49:22.785 INFO:teuthology.orchestra.run.smithi052.stdout:balancer 2023-12-16T20:49:22.785 INFO:teuthology.orchestra.run.smithi052.stdout:standby_count_wanted 0 2023-12-16T20:49:22.785 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:49:22.786 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:49:22.786 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.781+0000 7fbc36037700 1 -- 172.21.15.52:0/3051817235 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fbc18068f60 msgr2=0x7fbc1806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:22.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.781+0000 7fbc36037700 1 --2- 172.21.15.52:0/3051817235 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fbc18068f60 0x7fbc1806b410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbc20004560 tx=0x7fbc20009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:22.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.781+0000 7fbc36037700 1 -- 172.21.15.52:0/3051817235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc300fcbf0 msgr2=0x7fbc30071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:22.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.781+0000 7fbc36037700 1 --2- 172.21.15.52:0/3051817235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc300fcbf0 0x7fbc30071b10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400c8b0 tx=0x7fbc24004470 comp rx=0 tx=0).stop 2023-12-16T20:49:22.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.781+0000 7fbc36037700 1 -- 172.21.15.52:0/3051817235 shutdown_connections 2023-12-16T20:49:22.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.781+0000 7fbc36037700 1 --2- 172.21.15.52:0/3051817235 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fbc18068f60 0x7fbc1806b410 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:22.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.781+0000 7fbc36037700 1 --2- 172.21.15.52:0/3051817235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc300fcbf0 0x7fbc30071b10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:22.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.781+0000 7fbc36037700 1 --2- 172.21.15.52:0/3051817235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc300fde00 0x7fbc3006ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:22.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.781+0000 7fbc36037700 1 -- 172.21.15.52:0/3051817235 >> 172.21.15.52:0/3051817235 conn(0x7fbc300f82d0 msgr2=0x7fbc30101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:22.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.781+0000 7fbc36037700 1 -- 172.21.15.52:0/3051817235 shutdown_connections 2023-12-16T20:49:22.787 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:22.781+0000 7fbc36037700 1 -- 172.21.15.52:0/3051817235 wait complete. 2023-12-16T20:49:22.787 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 2 2023-12-16T20:49:23.144 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T20:49:23.156 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi052.front.sepia.ceph.com 2023-12-16T20:49:23.157 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 2' 2023-12-16T20:49:23.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:23 smithi052 ceph-mon[80761]: pgmap v121: 65 pgs: 55 unknown, 10 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:23.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:23 smithi052 ceph-mon[80761]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2023-12-16T20:49:23.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:23 smithi052 ceph-mon[80761]: osdmap e42: 6 total, 6 up, 6 in 2023-12-16T20:49:23.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:23.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:23.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:23.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.knuivd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T20:49:23.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.knuivd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T20:49:23.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:23 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:23.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:23 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3051817235' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-16T20:49:23.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:23 smithi150 ceph-mon[88000]: pgmap v121: 65 pgs: 55 unknown, 10 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:23.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:23 smithi150 ceph-mon[88000]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2023-12-16T20:49:23.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:23 smithi150 ceph-mon[88000]: osdmap e42: 6 total, 6 up, 6 in 2023-12-16T20:49:23.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:23 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:23.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:23 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:23.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:23 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:23.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:23 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.knuivd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T20:49:23.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:23 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.knuivd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T20:49:23.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:23 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:23.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:23 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/3051817235' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-16T20:49:23.427 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:24.181 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:24 smithi052 ceph-mon[80761]: Deploying daemon mds.cephfs.smithi150.knuivd on smithi150 2023-12-16T20:49:24.181 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:24 smithi052 ceph-mon[80761]: mds.? [v2:172.21.15.52:6826/1882583859,v1:172.21.15.52:6827/1882583859] up:boot 2023-12-16T20:49:24.181 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:24 smithi052 ceph-mon[80761]: daemon mds.cephfs.smithi052.bvmosa assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2023-12-16T20:49:24.181 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:24 smithi052 ceph-mon[80761]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-12-16T20:49:24.181 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:24 smithi052 ceph-mon[80761]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-12-16T20:49:24.182 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:24 smithi052 ceph-mon[80761]: Cluster is now healthy 2023-12-16T20:49:24.182 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:24 smithi052 ceph-mon[80761]: fsmap cephfs:0 1 up:standby 2023-12-16T20:49:24.182 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:24 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi052.bvmosa"}]: dispatch 2023-12-16T20:49:24.182 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:24 smithi052 ceph-mon[80761]: fsmap cephfs:1 {0=cephfs.smithi052.bvmosa=up:creating} 2023-12-16T20:49:24.182 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:24 smithi052 ceph-mon[80761]: daemon mds.cephfs.smithi052.bvmosa is now active in filesystem cephfs as rank 0 2023-12-16T20:49:24.363 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.360+0000 7f449a7b4700 1 -- 172.21.15.52:0/3910384359 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44940fcd00 msgr2=0x7f44940fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:24.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.360+0000 7f449a7b4700 1 --2- 172.21.15.52:0/3910384359 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44940fcd00 0x7f44940fd150 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4488004940 tx=0x7f4488009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:24.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.360+0000 7f449a7b4700 1 -- 172.21.15.52:0/3910384359 shutdown_connections 2023-12-16T20:49:24.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.360+0000 7f449a7b4700 1 --2- 172.21.15.52:0/3910384359 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44940fcd00 0x7f44940fd150 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:24.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.360+0000 7f449a7b4700 1 --2- 172.21.15.52:0/3910384359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44940fbaf0 0x7f44940fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:24.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.360+0000 7f449a7b4700 1 -- 172.21.15.52:0/3910384359 >> 172.21.15.52:0/3910384359 conn(0x7f44940f7190 msgr2=0x7f44940f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:24.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.360+0000 7f449a7b4700 1 -- 172.21.15.52:0/3910384359 shutdown_connections 2023-12-16T20:49:24.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.360+0000 7f449a7b4700 1 -- 172.21.15.52:0/3910384359 wait complete. 2023-12-16T20:49:24.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.361+0000 7f449a7b4700 1 Processor -- start 2023-12-16T20:49:24.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.361+0000 7f449a7b4700 1 -- start start 2023-12-16T20:49:24.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.361+0000 7f449a7b4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44940fbaf0 0x7f449410c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:24.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.361+0000 7f449a7b4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44940fcd00 0x7f449410c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:24.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.361+0000 7f449a7b4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449410cfb0 con 0x7f44940fcd00 2023-12-16T20:49:24.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.361+0000 7f449a7b4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449410d850 con 0x7f44940fbaf0 2023-12-16T20:49:24.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.361+0000 7f44937fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44940fcd00 0x7f449410c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:24.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.361+0000 7f44937fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44940fcd00 0x7f449410c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41318/0 (socket says 172.21.15.52:41318) 2023-12-16T20:49:24.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.361+0000 7f44937fe700 1 -- 172.21.15.52:0/2623790192 learned_addr learned my addr 172.21.15.52:0/2623790192 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:24.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.361+0000 7f4493fff700 1 --2- 172.21.15.52:0/2623790192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44940fbaf0 0x7f449410c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:24.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.361+0000 7f44937fe700 1 -- 172.21.15.52:0/2623790192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44940fbaf0 msgr2=0x7f449410c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:24.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.361+0000 7f44937fe700 1 --2- 172.21.15.52:0/2623790192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44940fbaf0 0x7f449410c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:24.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.362+0000 7f44937fe700 1 -- 172.21.15.52:0/2623790192 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4488005040 con 0x7f44940fcd00 2023-12-16T20:49:24.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.362+0000 7f4493fff700 1 --2- 172.21.15.52:0/2623790192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44940fbaf0 0x7f449410c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T20:49:24.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.362+0000 7f44937fe700 1 --2- 172.21.15.52:0/2623790192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44940fcd00 0x7f449410c9b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4488009360 tx=0x7f4488009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:24.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.362+0000 7f4490ff9700 1 -- 172.21.15.52:0/2623790192 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f448800e040 con 0x7f44940fcd00 2023-12-16T20:49:24.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.362+0000 7f449a7b4700 1 -- 172.21.15.52:0/2623790192 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449410dad0 con 0x7f44940fcd00 2023-12-16T20:49:24.366 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.362+0000 7f4490ff9700 1 -- 172.21.15.52:0/2623790192 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f448800bcd0 con 0x7f44940fcd00 2023-12-16T20:49:24.366 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.362+0000 7f4490ff9700 1 -- 172.21.15.52:0/2623790192 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f448801c630 con 0x7f44940fcd00 2023-12-16T20:49:24.366 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.362+0000 7f449a7b4700 1 -- 172.21.15.52:0/2623790192 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f449410e010 con 0x7f44940fcd00 2023-12-16T20:49:24.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.363+0000 7f449a7b4700 1 -- 172.21.15.52:0/2623790192 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44800052e0 con 0x7f44940fcd00 2023-12-16T20:49:24.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.364+0000 7f4490ff9700 1 -- 172.21.15.52:0/2623790192 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f4488013420 con 0x7f44940fcd00 2023-12-16T20:49:24.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.364+0000 7f4490ff9700 1 --2- 172.21.15.52:0/2623790192 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f447c068f60 0x7f447c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:24.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.364+0000 7f4490ff9700 1 -- 172.21.15.52:0/2623790192 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f4488088150 con 0x7f44940fcd00 2023-12-16T20:49:24.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.364+0000 7f4493fff700 1 --2- 172.21.15.52:0/2623790192 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f447c068f60 0x7f447c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:24.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.364+0000 7f4493fff700 1 --2- 172.21.15.52:0/2623790192 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f447c068f60 0x7f447c06b410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4484009fd0 tx=0x7f4484009f00 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:24.374 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.372+0000 7f4490ff9700 1 -- 172.21.15.52:0/2623790192 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f44880580b0 con 0x7f44940fcd00 2023-12-16T20:49:24.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:24 smithi150 ceph-mon[88000]: Deploying daemon mds.cephfs.smithi150.knuivd on smithi150 2023-12-16T20:49:24.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:24 smithi150 ceph-mon[88000]: mds.? [v2:172.21.15.52:6826/1882583859,v1:172.21.15.52:6827/1882583859] up:boot 2023-12-16T20:49:24.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:24 smithi150 ceph-mon[88000]: daemon mds.cephfs.smithi052.bvmosa assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2023-12-16T20:49:24.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:24 smithi150 ceph-mon[88000]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-12-16T20:49:24.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:24 smithi150 ceph-mon[88000]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-12-16T20:49:24.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:24 smithi150 ceph-mon[88000]: Cluster is now healthy 2023-12-16T20:49:24.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:24 smithi150 ceph-mon[88000]: fsmap cephfs:0 1 up:standby 2023-12-16T20:49:24.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:24 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi052.bvmosa"}]: dispatch 2023-12-16T20:49:24.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:24 smithi150 ceph-mon[88000]: fsmap cephfs:1 {0=cephfs.smithi052.bvmosa=up:creating} 2023-12-16T20:49:24.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:24 smithi150 ceph-mon[88000]: daemon mds.cephfs.smithi052.bvmosa is now active in filesystem cephfs as rank 0 2023-12-16T20:49:24.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:24.646+0000 7f449a7b4700 1 -- 172.21.15.52:0/2623790192 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f4480005b10 con 0x7f44940fcd00 2023-12-16T20:49:25.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:25 smithi052 ceph-mon[80761]: pgmap v123: 65 pgs: 22 unknown, 43 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:25.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:25 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:25 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:25 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:25 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.szlsle", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T20:49:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:25 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.szlsle", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T20:49:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:25 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:25 smithi052 ceph-mon[80761]: mds.? [v2:172.21.15.52:6826/1882583859,v1:172.21.15.52:6827/1882583859] up:active 2023-12-16T20:49:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:25 smithi052 ceph-mon[80761]: mds.? [v2:172.21.15.150:6824/351844450,v1:172.21.15.150:6825/351844450] up:boot 2023-12-16T20:49:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:25 smithi052 ceph-mon[80761]: fsmap cephfs:1 {0=cephfs.smithi052.bvmosa=up:active} 1 up:standby 2023-12-16T20:49:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:25 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi150.knuivd"}]: dispatch 2023-12-16T20:49:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:25 smithi052 ceph-mon[80761]: fsmap cephfs:1 {0=cephfs.smithi052.bvmosa=up:active} 1 up:standby 2023-12-16T20:49:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:25 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/2623790192' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2023-12-16T20:49:25.367 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:25 smithi150 ceph-mon[88000]: pgmap v123: 65 pgs: 22 unknown, 43 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T20:49:25.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:25 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:25.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:25 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:25.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:25 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:25.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:25 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.szlsle", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T20:49:25.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:25 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.szlsle", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T20:49:25.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:25 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:25.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:25 smithi150 ceph-mon[88000]: mds.? [v2:172.21.15.52:6826/1882583859,v1:172.21.15.52:6827/1882583859] up:active 2023-12-16T20:49:25.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:25 smithi150 ceph-mon[88000]: mds.? [v2:172.21.15.150:6824/351844450,v1:172.21.15.150:6825/351844450] up:boot 2023-12-16T20:49:25.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:25 smithi150 ceph-mon[88000]: fsmap cephfs:1 {0=cephfs.smithi052.bvmosa=up:active} 1 up:standby 2023-12-16T20:49:25.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:25 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi150.knuivd"}]: dispatch 2023-12-16T20:49:25.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:25 smithi150 ceph-mon[88000]: fsmap cephfs:1 {0=cephfs.smithi052.bvmosa=up:active} 1 up:standby 2023-12-16T20:49:25.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:25 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/2623790192' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2023-12-16T20:49:25.479 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:25.475+0000 7f4490ff9700 1 -- 172.21.15.52:0/2623790192 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v7) v1 ==== 105+0+0 (secure 0 0 0) 0x7f4488057c40 con 0x7f44940fcd00 2023-12-16T20:49:25.480 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:25.478+0000 7f449a7b4700 1 -- 172.21.15.52:0/2623790192 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f447c068f60 msgr2=0x7f447c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:25.480 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:25.478+0000 7f449a7b4700 1 --2- 172.21.15.52:0/2623790192 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f447c068f60 0x7f447c06b410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4484009fd0 tx=0x7f4484009f00 comp rx=0 tx=0).stop 2023-12-16T20:49:25.480 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:25.478+0000 7f449a7b4700 1 -- 172.21.15.52:0/2623790192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44940fcd00 msgr2=0x7f449410c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:25.480 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:25.478+0000 7f449a7b4700 1 --2- 172.21.15.52:0/2623790192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44940fcd00 0x7f449410c9b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4488009360 tx=0x7f4488009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:25.480 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:25.478+0000 7f449a7b4700 1 -- 172.21.15.52:0/2623790192 shutdown_connections 2023-12-16T20:49:25.480 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:25.478+0000 7f449a7b4700 1 --2- 172.21.15.52:0/2623790192 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f447c068f60 0x7f447c06b410 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:25.480 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:25.478+0000 7f449a7b4700 1 --2- 172.21.15.52:0/2623790192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44940fbaf0 0x7f449410c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:25.480 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:25.478+0000 7f449a7b4700 1 --2- 172.21.15.52:0/2623790192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44940fcd00 0x7f449410c9b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:25.480 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:25.478+0000 7f449a7b4700 1 -- 172.21.15.52:0/2623790192 >> 172.21.15.52:0/2623790192 conn(0x7f44940f7190 msgr2=0x7f44940fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:25.480 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:25.478+0000 7f449a7b4700 1 -- 172.21.15.52:0/2623790192 shutdown_connections 2023-12-16T20:49:25.480 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:25.478+0000 7f449a7b4700 1 -- 172.21.15.52:0/2623790192 wait complete. 2023-12-16T20:49:25.810 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T20:49:25.820 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi052.front.sepia.ceph.com 2023-12-16T20:49:25.820 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2023-12-16T20:49:26.103 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:26.195 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:26 smithi052 ceph-mon[80761]: Deploying daemon mds.cephfs.smithi052.szlsle on smithi052 2023-12-16T20:49:26.195 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:26 smithi052 ceph-mon[80761]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-12-16T20:49:26.195 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:26 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/2623790192' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2023-12-16T20:49:26.195 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:26 smithi052 ceph-mon[80761]: daemon mds.cephfs.smithi150.knuivd assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-12-16T20:49:26.195 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:26 smithi052 ceph-mon[80761]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2023-12-16T20:49:26.195 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:26 smithi052 ceph-mon[80761]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-12-16T20:49:26.195 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:26 smithi052 ceph-mon[80761]: fsmap cephfs:1 {0=cephfs.smithi052.bvmosa=up:active} 1 up:standby 2023-12-16T20:49:26.195 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:26 smithi052 ceph-mon[80761]: fsmap cephfs:2 {0=cephfs.smithi052.bvmosa=up:active,1=cephfs.smithi150.knuivd=up:creating} 2023-12-16T20:49:26.196 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:26 smithi052 ceph-mon[80761]: daemon mds.cephfs.smithi150.knuivd is now active in filesystem cephfs as rank 1 2023-12-16T20:49:26.196 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:26.196 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:26.196 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:26.196 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.njrgli", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T20:49:26.196 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.njrgli", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T20:49:26.196 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:26 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:26.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:26 smithi150 ceph-mon[88000]: Deploying daemon mds.cephfs.smithi052.szlsle on smithi052 2023-12-16T20:49:26.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:26 smithi150 ceph-mon[88000]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-12-16T20:49:26.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:26 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/2623790192' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2023-12-16T20:49:26.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:26 smithi150 ceph-mon[88000]: daemon mds.cephfs.smithi150.knuivd assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-12-16T20:49:26.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:26 smithi150 ceph-mon[88000]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2023-12-16T20:49:26.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:26 smithi150 ceph-mon[88000]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-12-16T20:49:26.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:26 smithi150 ceph-mon[88000]: fsmap cephfs:1 {0=cephfs.smithi052.bvmosa=up:active} 1 up:standby 2023-12-16T20:49:26.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:26 smithi150 ceph-mon[88000]: fsmap cephfs:2 {0=cephfs.smithi052.bvmosa=up:active,1=cephfs.smithi150.knuivd=up:creating} 2023-12-16T20:49:26.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:26 smithi150 ceph-mon[88000]: daemon mds.cephfs.smithi150.knuivd is now active in filesystem cephfs as rank 1 2023-12-16T20:49:26.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:26 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:26.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:26 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:26.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:26 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:26.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:26 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.njrgli", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T20:49:26.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:26 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.njrgli", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T20:49:26.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:26 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:26.944 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.940+0000 7fec96bc0700 1 -- 172.21.15.52:0/269666699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec900fcbf0 msgr2=0x7fec900fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:26.944 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.940+0000 7fec96bc0700 1 --2- 172.21.15.52:0/269666699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec900fcbf0 0x7fec900fd000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fec84004940 tx=0x7fec84009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:26.944 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.940+0000 7fec96bc0700 1 -- 172.21.15.52:0/269666699 shutdown_connections 2023-12-16T20:49:26.944 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.940+0000 7fec96bc0700 1 --2- 172.21.15.52:0/269666699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec900fde00 0x7fec900fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:26.944 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.940+0000 7fec96bc0700 1 --2- 172.21.15.52:0/269666699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec900fcbf0 0x7fec900fd000 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:26.944 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.940+0000 7fec96bc0700 1 -- 172.21.15.52:0/269666699 >> 172.21.15.52:0/269666699 conn(0x7fec900f82d0 msgr2=0x7fec900fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:26.944 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.940+0000 7fec96bc0700 1 -- 172.21.15.52:0/269666699 shutdown_connections 2023-12-16T20:49:26.944 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.940+0000 7fec96bc0700 1 -- 172.21.15.52:0/269666699 wait complete. 2023-12-16T20:49:26.944 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.941+0000 7fec96bc0700 1 Processor -- start 2023-12-16T20:49:26.945 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.941+0000 7fec96bc0700 1 -- start start 2023-12-16T20:49:26.945 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.941+0000 7fec96bc0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec900fcbf0 0x7fec9010d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:26.945 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.941+0000 7fec96bc0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec900fde00 0x7fec9010da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:26.945 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.941+0000 7fec96bc0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9010e010 con 0x7fec900fcbf0 2023-12-16T20:49:26.945 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.941+0000 7fec96bc0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9010e8b0 con 0x7fec900fde00 2023-12-16T20:49:26.945 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.941+0000 7fec9495c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec900fcbf0 0x7fec9010d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:26.945 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.941+0000 7fec9495c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec900fcbf0 0x7fec9010d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41360/0 (socket says 172.21.15.52:41360) 2023-12-16T20:49:26.945 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.941+0000 7fec9495c700 1 -- 172.21.15.52:0/488682944 learned_addr learned my addr 172.21.15.52:0/488682944 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:26.945 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.941+0000 7fec8ffff700 1 --2- 172.21.15.52:0/488682944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec900fde00 0x7fec9010da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:26.945 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.941+0000 7fec9495c700 1 -- 172.21.15.52:0/488682944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec900fde00 msgr2=0x7fec9010da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:26.945 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.941+0000 7fec9495c700 1 --2- 172.21.15.52:0/488682944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec900fde00 0x7fec9010da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:26.945 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.941+0000 7fec9495c700 1 -- 172.21.15.52:0/488682944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec84005040 con 0x7fec900fcbf0 2023-12-16T20:49:26.946 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.942+0000 7fec9495c700 1 --2- 172.21.15.52:0/488682944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec900fcbf0 0x7fec9010d4d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fec84000c00 tx=0x7fec84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:26.946 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.942+0000 7fec8d7fa700 1 -- 172.21.15.52:0/488682944 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec84021070 con 0x7fec900fcbf0 2023-12-16T20:49:26.946 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.942+0000 7fec8d7fa700 1 -- 172.21.15.52:0/488682944 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec8401c400 con 0x7fec900fcbf0 2023-12-16T20:49:26.946 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.942+0000 7fec96bc0700 1 -- 172.21.15.52:0/488682944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec9010eb30 con 0x7fec900fcbf0 2023-12-16T20:49:26.946 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.942+0000 7fec8d7fa700 1 -- 172.21.15.52:0/488682944 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec8400e070 con 0x7fec900fcbf0 2023-12-16T20:49:26.946 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.942+0000 7fec96bc0700 1 -- 172.21.15.52:0/488682944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec9010f010 con 0x7fec900fcbf0 2023-12-16T20:49:26.947 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.943+0000 7fec96bc0700 1 -- 172.21.15.52:0/488682944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9004bb80 con 0x7fec900fcbf0 2023-12-16T20:49:26.947 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.943+0000 7fec8d7fa700 1 -- 172.21.15.52:0/488682944 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fec8401ca00 con 0x7fec900fcbf0 2023-12-16T20:49:26.947 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.944+0000 7fec8d7fa700 1 --2- 172.21.15.52:0/488682944 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fec80068f60 0x7fec8006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:26.947 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.944+0000 7fec8d7fa700 1 -- 172.21.15.52:0/488682944 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fec84087f50 con 0x7fec900fcbf0 2023-12-16T20:49:26.947 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.944+0000 7fec8ffff700 1 --2- 172.21.15.52:0/488682944 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fec80068f60 0x7fec8006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:26.947 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.944+0000 7fec8ffff700 1 --2- 172.21.15.52:0/488682944 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fec80068f60 0x7fec8006b410 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009b80 tx=0x7fec7c00b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:26.950 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:26.949+0000 7fec8d7fa700 1 -- 172.21.15.52:0/488682944 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fec840b9a00 con 0x7fec900fcbf0 2023-12-16T20:49:27.190 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:27.188+0000 7fec96bc0700 1 -- 172.21.15.52:0/488682944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fec90105590 con 0x7fec900fcbf0 2023-12-16T20:49:27.191 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:27 smithi052 ceph-mon[80761]: pgmap v124: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 1 op/s 2023-12-16T20:49:27.191 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:27 smithi052 ceph-mon[80761]: Deploying daemon mds.cephfs.smithi150.njrgli on smithi150 2023-12-16T20:49:27.191 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:27 smithi052 ceph-mon[80761]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2023-12-16T20:49:27.191 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:27 smithi052 ceph-mon[80761]: Cluster is now healthy 2023-12-16T20:49:27.191 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:27 smithi052 ceph-mon[80761]: mds.? [v2:172.21.15.150:6824/351844450,v1:172.21.15.150:6825/351844450] up:active 2023-12-16T20:49:27.191 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:27 smithi052 ceph-mon[80761]: mds.? [v2:172.21.15.52:6828/1781554610,v1:172.21.15.52:6829/1781554610] up:boot 2023-12-16T20:49:27.191 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:27 smithi052 ceph-mon[80761]: fsmap cephfs:2 {0=cephfs.smithi052.bvmosa=up:active,1=cephfs.smithi150.knuivd=up:active} 1 up:standby 2023-12-16T20:49:27.191 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:27 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi052.szlsle"}]: dispatch 2023-12-16T20:49:27.367 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:27 smithi150 ceph-mon[88000]: pgmap v124: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 1 op/s 2023-12-16T20:49:27.367 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:27 smithi150 ceph-mon[88000]: Deploying daemon mds.cephfs.smithi150.njrgli on smithi150 2023-12-16T20:49:27.367 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:27 smithi150 ceph-mon[88000]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2023-12-16T20:49:27.367 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:27 smithi150 ceph-mon[88000]: Cluster is now healthy 2023-12-16T20:49:27.367 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:27 smithi150 ceph-mon[88000]: mds.? [v2:172.21.15.150:6824/351844450,v1:172.21.15.150:6825/351844450] up:active 2023-12-16T20:49:27.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:27 smithi150 ceph-mon[88000]: mds.? [v2:172.21.15.52:6828/1781554610,v1:172.21.15.52:6829/1781554610] up:boot 2023-12-16T20:49:27.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:27 smithi150 ceph-mon[88000]: fsmap cephfs:2 {0=cephfs.smithi052.bvmosa=up:active,1=cephfs.smithi150.knuivd=up:active} 1 up:standby 2023-12-16T20:49:27.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:27 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi052.szlsle"}]: dispatch 2023-12-16T20:49:27.446 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:27.446+0000 7fec8d7fa700 1 -- 172.21.15.52:0/488682944 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v10) v1 ==== 121+0+0 (secure 0 0 0) 0x7fec84057eb0 con 0x7fec900fcbf0 2023-12-16T20:49:27.450 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:27.449+0000 7fec96bc0700 1 -- 172.21.15.52:0/488682944 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fec80068f60 msgr2=0x7fec8006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:27.450 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:27.449+0000 7fec96bc0700 1 --2- 172.21.15.52:0/488682944 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fec80068f60 0x7fec8006b410 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009b80 tx=0x7fec7c00b040 comp rx=0 tx=0).stop 2023-12-16T20:49:27.450 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:27.449+0000 7fec96bc0700 1 -- 172.21.15.52:0/488682944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec900fcbf0 msgr2=0x7fec9010d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:27.450 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:27.449+0000 7fec96bc0700 1 --2- 172.21.15.52:0/488682944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec900fcbf0 0x7fec9010d4d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fec84000c00 tx=0x7fec84009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:27.450 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:27.449+0000 7fec96bc0700 1 -- 172.21.15.52:0/488682944 shutdown_connections 2023-12-16T20:49:27.450 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:27.449+0000 7fec96bc0700 1 --2- 172.21.15.52:0/488682944 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fec80068f60 0x7fec8006b410 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:27.450 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:27.449+0000 7fec96bc0700 1 --2- 172.21.15.52:0/488682944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec900fcbf0 0x7fec9010d4d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:27.450 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:27.449+0000 7fec96bc0700 1 --2- 172.21.15.52:0/488682944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec900fde00 0x7fec9010da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:27.450 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:27.449+0000 7fec96bc0700 1 -- 172.21.15.52:0/488682944 >> 172.21.15.52:0/488682944 conn(0x7fec900f82d0 msgr2=0x7fec90101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:27.450 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:27.449+0000 7fec96bc0700 1 -- 172.21.15.52:0/488682944 shutdown_connections 2023-12-16T20:49:27.450 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:27.449+0000 7fec96bc0700 1 -- 172.21.15.52:0/488682944 wait complete. 2023-12-16T20:49:27.825 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T20:49:27.836 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi052.front.sepia.ceph.com 2023-12-16T20:49:27.836 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2023-12-16T20:49:28.441 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:28 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/488682944' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-12-16T20:49:28.441 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:28 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/488682944' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-12-16T20:49:28.441 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:28 smithi150 ceph-mon[88000]: fsmap cephfs:2 {0=cephfs.smithi052.bvmosa=up:active,1=cephfs.smithi150.knuivd=up:active} 1 up:standby-replay 2023-12-16T20:49:28.441 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:28 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:28.441 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:28 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:28.441 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:28 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:28.441 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:28 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:28.441 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:28 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:28.441 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:28 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:49:28.441 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:28 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:28.469 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:28 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/488682944' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-12-16T20:49:28.469 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:28 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/488682944' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-12-16T20:49:28.469 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:28 smithi052 ceph-mon[80761]: fsmap cephfs:2 {0=cephfs.smithi052.bvmosa=up:active,1=cephfs.smithi150.knuivd=up:active} 1 up:standby-replay 2023-12-16T20:49:28.469 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:28 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:28.469 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:28 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:28.469 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:28 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:28.469 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:28 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:28.469 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:28 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:28.469 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:28 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:49:28.469 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:28 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:28.502 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:29.256 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:29 smithi052 ceph-mon[80761]: pgmap v125: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 685 B/s rd, 1.5 KiB/s wr, 8 op/s 2023-12-16T20:49:29.256 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:29 smithi052 ceph-mon[80761]: mds.? [v2:172.21.15.150:6826/4088837116,v1:172.21.15.150:6827/4088837116] up:boot 2023-12-16T20:49:29.256 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:29 smithi052 ceph-mon[80761]: fsmap cephfs:2 {0=cephfs.smithi052.bvmosa=up:active,1=cephfs.smithi150.knuivd=up:active} 1 up:standby-replay 1 up:standby 2023-12-16T20:49:29.256 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:29 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi150.njrgli"}]: dispatch 2023-12-16T20:49:29.256 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:29 smithi052 ceph-mon[80761]: fsmap cephfs:2 {0=cephfs.smithi052.bvmosa=up:active,1=cephfs.smithi150.knuivd=up:active} 2 up:standby-replay 2023-12-16T20:49:29.256 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:29 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:29.256 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:29 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:29.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:29 smithi150 ceph-mon[88000]: pgmap v125: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 685 B/s rd, 1.5 KiB/s wr, 8 op/s 2023-12-16T20:49:29.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:29 smithi150 ceph-mon[88000]: mds.? [v2:172.21.15.150:6826/4088837116,v1:172.21.15.150:6827/4088837116] up:boot 2023-12-16T20:49:29.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:29 smithi150 ceph-mon[88000]: fsmap cephfs:2 {0=cephfs.smithi052.bvmosa=up:active,1=cephfs.smithi150.knuivd=up:active} 1 up:standby-replay 1 up:standby 2023-12-16T20:49:29.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:29 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi150.njrgli"}]: dispatch 2023-12-16T20:49:29.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:29 smithi150 ceph-mon[88000]: fsmap cephfs:2 {0=cephfs.smithi052.bvmosa=up:active,1=cephfs.smithi150.knuivd=up:active} 2 up:standby-replay 2023-12-16T20:49:29.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:29 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:29.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:29 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:29.520 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.518+0000 7fe354167700 1 -- 172.21.15.52:0/307192912 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe34c0fbaf0 msgr2=0x7fe34c0fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:29.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.518+0000 7fe354167700 1 --2- 172.21.15.52:0/307192912 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe34c0fbaf0 0x7fe34c0fbf00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe348004940 tx=0x7fe348009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:29.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.518+0000 7fe354167700 1 -- 172.21.15.52:0/307192912 shutdown_connections 2023-12-16T20:49:29.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.518+0000 7fe354167700 1 --2- 172.21.15.52:0/307192912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe34c0fcd00 0x7fe34c0fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:29.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.518+0000 7fe354167700 1 --2- 172.21.15.52:0/307192912 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe34c0fbaf0 0x7fe34c0fbf00 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:29.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.518+0000 7fe354167700 1 -- 172.21.15.52:0/307192912 >> 172.21.15.52:0/307192912 conn(0x7fe34c0f7190 msgr2=0x7fe34c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:29.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.518+0000 7fe354167700 1 -- 172.21.15.52:0/307192912 shutdown_connections 2023-12-16T20:49:29.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.518+0000 7fe354167700 1 -- 172.21.15.52:0/307192912 wait complete. 2023-12-16T20:49:29.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.519+0000 7fe354167700 1 Processor -- start 2023-12-16T20:49:29.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.519+0000 7fe354167700 1 -- start start 2023-12-16T20:49:29.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.519+0000 7fe354167700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe34c0fbaf0 0x7fe34c10c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:29.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.519+0000 7fe354167700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe34c0fcd00 0x7fe34c10c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:29.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.519+0000 7fe354167700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c10cfc0 con 0x7fe34c0fbaf0 2023-12-16T20:49:29.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.519+0000 7fe354167700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c10d860 con 0x7fe34c0fcd00 2023-12-16T20:49:29.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.519+0000 7fe351f03700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe34c0fbaf0 0x7fe34c10c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:29.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.519+0000 7fe351f03700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe34c0fbaf0 0x7fe34c10c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41392/0 (socket says 172.21.15.52:41392) 2023-12-16T20:49:29.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.519+0000 7fe351f03700 1 -- 172.21.15.52:0/912511558 learned_addr learned my addr 172.21.15.52:0/912511558 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:29.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.519+0000 7fe351702700 1 --2- 172.21.15.52:0/912511558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe34c0fcd00 0x7fe34c10c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:29.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.519+0000 7fe351f03700 1 -- 172.21.15.52:0/912511558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe34c0fcd00 msgr2=0x7fe34c10c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:29.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.519+0000 7fe351f03700 1 --2- 172.21.15.52:0/912511558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe34c0fcd00 0x7fe34c10c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:29.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.519+0000 7fe351f03700 1 -- 172.21.15.52:0/912511558 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe348005040 con 0x7fe34c0fbaf0 2023-12-16T20:49:29.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.519+0000 7fe351f03700 1 --2- 172.21.15.52:0/912511558 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe34c0fbaf0 0x7fe34c10c480 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe348000c00 tx=0x7fe348009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:29.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.520+0000 7fe3427fc700 1 -- 172.21.15.52:0/912511558 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe34800e040 con 0x7fe34c0fbaf0 2023-12-16T20:49:29.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.520+0000 7fe3427fc700 1 -- 172.21.15.52:0/912511558 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe348009af0 con 0x7fe34c0fbaf0 2023-12-16T20:49:29.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.520+0000 7fe3427fc700 1 -- 172.21.15.52:0/912511558 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe34801c630 con 0x7fe34c0fbaf0 2023-12-16T20:49:29.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.520+0000 7fe354167700 1 -- 172.21.15.52:0/912511558 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34c10dae0 con 0x7fe34c0fbaf0 2023-12-16T20:49:29.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.520+0000 7fe354167700 1 -- 172.21.15.52:0/912511558 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34c10e020 con 0x7fe34c0fbaf0 2023-12-16T20:49:29.523 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.521+0000 7fe354167700 1 -- 172.21.15.52:0/912511558 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34c04bb80 con 0x7fe34c0fbaf0 2023-12-16T20:49:29.523 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.521+0000 7fe3427fc700 1 -- 172.21.15.52:0/912511558 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe34800bc50 con 0x7fe34c0fbaf0 2023-12-16T20:49:29.523 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.522+0000 7fe3427fc700 1 --2- 172.21.15.52:0/912511558 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe338068f60 0x7fe33806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:29.523 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.522+0000 7fe3427fc700 1 -- 172.21.15.52:0/912511558 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe348087fd0 con 0x7fe34c0fbaf0 2023-12-16T20:49:29.523 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.522+0000 7fe351702700 1 --2- 172.21.15.52:0/912511558 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe338068f60 0x7fe33806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:29.524 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.522+0000 7fe351702700 1 --2- 172.21.15.52:0/912511558 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe338068f60 0x7fe33806b410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe33c004f40 tx=0x7fe33c00b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:29.527 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.527+0000 7fe3427fc700 1 -- 172.21.15.52:0/912511558 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe348057fd0 con 0x7fe34c0fbaf0 2023-12-16T20:49:29.791 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:29.790+0000 7fe354167700 1 -- 172.21.15.52:0/912511558 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe34c104490 con 0x7fe34c0fbaf0 2023-12-16T20:49:30.121 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:30.120+0000 7fe3427fc700 1 -- 172.21.15.52:0/912511558 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v13) v1 ==== 168+0+0 (secure 0 0 0) 0x7fe348057b60 con 0x7fe34c0fbaf0 2023-12-16T20:49:30.124 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:30.123+0000 7fe354167700 1 -- 172.21.15.52:0/912511558 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe338068f60 msgr2=0x7fe33806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:30.124 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:30.123+0000 7fe354167700 1 --2- 172.21.15.52:0/912511558 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe338068f60 0x7fe33806b410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe33c004f40 tx=0x7fe33c00b040 comp rx=0 tx=0).stop 2023-12-16T20:49:30.124 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:30.123+0000 7fe354167700 1 -- 172.21.15.52:0/912511558 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe34c0fbaf0 msgr2=0x7fe34c10c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:30.124 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:30.123+0000 7fe354167700 1 --2- 172.21.15.52:0/912511558 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe34c0fbaf0 0x7fe34c10c480 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe348000c00 tx=0x7fe348009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:30.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:30.124+0000 7fe354167700 1 -- 172.21.15.52:0/912511558 shutdown_connections 2023-12-16T20:49:30.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:30.124+0000 7fe354167700 1 --2- 172.21.15.52:0/912511558 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe338068f60 0x7fe33806b410 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:30.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:30.124+0000 7fe354167700 1 --2- 172.21.15.52:0/912511558 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe34c0fbaf0 0x7fe34c10c480 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:30.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:30.124+0000 7fe354167700 1 --2- 172.21.15.52:0/912511558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe34c0fcd00 0x7fe34c10c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:30.125 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:30.124+0000 7fe354167700 1 -- 172.21.15.52:0/912511558 >> 172.21.15.52:0/912511558 conn(0x7fe34c0f7190 msgr2=0x7fe34c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:30.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:30.125+0000 7fe354167700 1 -- 172.21.15.52:0/912511558 shutdown_connections 2023-12-16T20:49:30.126 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:30.125+0000 7fe354167700 1 -- 172.21.15.52:0/912511558 wait complete. 2023-12-16T20:49:30.127 INFO:teuthology.orchestra.run.smithi052.stderr:inline data enabled 2023-12-16T20:49:30.419 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:30 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/912511558' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2023-12-16T20:49:30.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:30 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/912511558' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2023-12-16T20:49:30.474 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T20:49:30.484 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi052.front.sepia.ceph.com 2023-12-16T20:49:30.484 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- bash -c 'ceph fs dump' 2023-12-16T20:49:31.083 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:31.394 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:31 smithi052 ceph-mon[80761]: pgmap v126: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2023-12-16T20:49:31.394 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:31 smithi052 ceph-mon[80761]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2023-12-16T20:49:31.394 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:31 smithi052 ceph-mon[80761]: mds.? [v2:172.21.15.150:6824/351844450,v1:172.21.15.150:6825/351844450] up:active 2023-12-16T20:49:31.395 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:31 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/912511558' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2023-12-16T20:49:31.395 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:31 smithi052 ceph-mon[80761]: mds.? [v2:172.21.15.52:6828/1781554610,v1:172.21.15.52:6829/1781554610] up:standby-replay 2023-12-16T20:49:31.395 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:31 smithi052 ceph-mon[80761]: Dropping low affinity active daemon mds.cephfs.smithi052.bvmosa in favor of higher affinity standby. 2023-12-16T20:49:31.395 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:31 smithi052 ceph-mon[80761]: Replacing daemon mds.cephfs.smithi052.bvmosa as rank 0 with standby daemon mds.cephfs.smithi052.szlsle 2023-12-16T20:49:31.395 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:31 smithi052 ceph-mon[80761]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-12-16T20:49:31.395 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:31 smithi052 ceph-mon[80761]: fsmap cephfs:2 {0=cephfs.smithi052.bvmosa=up:active,1=cephfs.smithi150.knuivd=up:active} 2 up:standby-replay 2023-12-16T20:49:31.395 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:31 smithi052 ceph-mon[80761]: osdmap e43: 6 total, 6 up, 6 in 2023-12-16T20:49:31.395 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:31 smithi052 ceph-mon[80761]: fsmap cephfs:2/2 {0=cephfs.smithi052.szlsle=up:replay,1=cephfs.smithi150.knuivd=up:active} 1 up:standby-replay 2023-12-16T20:49:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:31 smithi150 ceph-mon[88000]: pgmap v126: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2023-12-16T20:49:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:31 smithi150 ceph-mon[88000]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2023-12-16T20:49:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:31 smithi150 ceph-mon[88000]: mds.? [v2:172.21.15.150:6824/351844450,v1:172.21.15.150:6825/351844450] up:active 2023-12-16T20:49:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:31 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/912511558' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2023-12-16T20:49:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:31 smithi150 ceph-mon[88000]: mds.? [v2:172.21.15.52:6828/1781554610,v1:172.21.15.52:6829/1781554610] up:standby-replay 2023-12-16T20:49:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:31 smithi150 ceph-mon[88000]: Dropping low affinity active daemon mds.cephfs.smithi052.bvmosa in favor of higher affinity standby. 2023-12-16T20:49:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:31 smithi150 ceph-mon[88000]: Replacing daemon mds.cephfs.smithi052.bvmosa as rank 0 with standby daemon mds.cephfs.smithi052.szlsle 2023-12-16T20:49:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:31 smithi150 ceph-mon[88000]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-12-16T20:49:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:31 smithi150 ceph-mon[88000]: fsmap cephfs:2 {0=cephfs.smithi052.bvmosa=up:active,1=cephfs.smithi150.knuivd=up:active} 2 up:standby-replay 2023-12-16T20:49:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:31 smithi150 ceph-mon[88000]: osdmap e43: 6 total, 6 up, 6 in 2023-12-16T20:49:31.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:31 smithi150 ceph-mon[88000]: fsmap cephfs:2/2 {0=cephfs.smithi052.szlsle=up:replay,1=cephfs.smithi150.knuivd=up:active} 1 up:standby-replay 2023-12-16T20:49:32.119 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.118+0000 7f6ab7ad0700 1 -- 172.21.15.52:0/2173258826 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ab00fbaf0 msgr2=0x7f6ab00fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:32.119 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.118+0000 7f6ab7ad0700 1 --2- 172.21.15.52:0/2173258826 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ab00fbaf0 0x7f6ab00fbf00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6aac004940 tx=0x7f6aac009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:32.119 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.118+0000 7f6ab7ad0700 1 -- 172.21.15.52:0/2173258826 shutdown_connections 2023-12-16T20:49:32.119 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.118+0000 7f6ab7ad0700 1 --2- 172.21.15.52:0/2173258826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab00fcd00 0x7f6ab00fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:32.120 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.118+0000 7f6ab7ad0700 1 --2- 172.21.15.52:0/2173258826 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ab00fbaf0 0x7f6ab00fbf00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:32.120 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.118+0000 7f6ab7ad0700 1 -- 172.21.15.52:0/2173258826 >> 172.21.15.52:0/2173258826 conn(0x7f6ab00f71d0 msgr2=0x7f6ab00f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:32.120 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.119+0000 7f6ab7ad0700 1 -- 172.21.15.52:0/2173258826 shutdown_connections 2023-12-16T20:49:32.120 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.119+0000 7f6ab7ad0700 1 -- 172.21.15.52:0/2173258826 wait complete. 2023-12-16T20:49:32.121 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.119+0000 7f6ab7ad0700 1 Processor -- start 2023-12-16T20:49:32.121 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.119+0000 7f6ab7ad0700 1 -- start start 2023-12-16T20:49:32.121 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.119+0000 7f6ab7ad0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab00fbaf0 0x7f6ab010f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:32.121 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.119+0000 7f6ab7ad0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ab00fcd00 0x7f6ab010fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:32.121 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.119+0000 7f6ab7ad0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab0110340 con 0x7f6ab00fcd00 2023-12-16T20:49:32.121 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.119+0000 7f6ab7ad0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab0110b80 con 0x7f6ab00fbaf0 2023-12-16T20:49:32.122 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.119+0000 7f6ab506b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ab00fcd00 0x7f6ab010fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:32.122 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.119+0000 7f6ab506b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ab00fcd00 0x7f6ab010fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41414/0 (socket says 172.21.15.52:41414) 2023-12-16T20:49:32.122 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.119+0000 7f6ab506b700 1 -- 172.21.15.52:0/387112917 learned_addr learned my addr 172.21.15.52:0/387112917 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:32.122 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.120+0000 7f6ab586c700 1 --2- 172.21.15.52:0/387112917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab00fbaf0 0x7f6ab010f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:32.122 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.120+0000 7f6ab586c700 1 -- 172.21.15.52:0/387112917 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ab00fcd00 msgr2=0x7f6ab010fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:32.122 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.120+0000 7f6ab586c700 1 --2- 172.21.15.52:0/387112917 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ab00fcd00 0x7f6ab010fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:32.122 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.120+0000 7f6ab586c700 1 -- 172.21.15.52:0/387112917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac005040 con 0x7f6ab00fbaf0 2023-12-16T20:49:32.122 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.120+0000 7f6ab586c700 1 --2- 172.21.15.52:0/387112917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab00fbaf0 0x7f6ab010f770 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6aac009fd0 tx=0x7f6aac009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:32.122 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.120+0000 7f6aa67fc700 1 -- 172.21.15.52:0/387112917 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6aac00e040 con 0x7f6ab00fbaf0 2023-12-16T20:49:32.122 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.120+0000 7f6ab7ad0700 1 -- 172.21.15.52:0/387112917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab0110e00 con 0x7f6ab00fbaf0 2023-12-16T20:49:32.122 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.120+0000 7f6aa67fc700 1 -- 172.21.15.52:0/387112917 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6aac004290 con 0x7f6ab00fbaf0 2023-12-16T20:49:32.122 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.120+0000 7f6aa67fc700 1 -- 172.21.15.52:0/387112917 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6aac01c3f0 con 0x7f6ab00fbaf0 2023-12-16T20:49:32.122 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.121+0000 7f6ab7ad0700 1 -- 172.21.15.52:0/387112917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab0111340 con 0x7f6ab00fbaf0 2023-12-16T20:49:32.123 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.122+0000 7f6ab7ad0700 1 -- 172.21.15.52:0/387112917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab004bb80 con 0x7f6ab00fbaf0 2023-12-16T20:49:32.123 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.122+0000 7f6aa67fc700 1 -- 172.21.15.52:0/387112917 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f6aac013420 con 0x7f6ab00fbaf0 2023-12-16T20:49:32.123 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.122+0000 7f6aa67fc700 1 --2- 172.21.15.52:0/387112917 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6a9c068f70 0x7f6a9c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:32.123 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.123+0000 7f6aa67fc700 1 -- 172.21.15.52:0/387112917 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6aac088200 con 0x7f6ab00fbaf0 2023-12-16T20:49:32.123 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.123+0000 7f6ab506b700 1 --2- 172.21.15.52:0/387112917 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6a9c068f70 0x7f6a9c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:32.124 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.123+0000 7f6ab506b700 1 --2- 172.21.15.52:0/387112917 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6a9c068f70 0x7f6a9c06b420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0009ca0 tx=0x7f6aa0009250 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:32.129 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.128+0000 7f6aa67fc700 1 -- 172.21.15.52:0/387112917 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6aac058070 con 0x7f6ab00fbaf0 2023-12-16T20:49:32.381 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.381+0000 7f6ab7ad0700 1 -- 172.21.15.52:0/387112917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6ab0111710 con 0x7f6ab00fbaf0 2023-12-16T20:49:32.382 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:32 smithi052 ceph-mon[80761]: mds.? [v2:172.21.15.52:6826/2170358701,v1:172.21.15.52:6827/2170358701] up:boot 2023-12-16T20:49:32.382 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:32 smithi052 ceph-mon[80761]: mds.? [v2:172.21.15.52:6828/1781554610,v1:172.21.15.52:6829/1781554610] up:resolve 2023-12-16T20:49:32.382 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:32 smithi052 ceph-mon[80761]: fsmap cephfs:2/2 {0=cephfs.smithi052.szlsle=up:resolve,1=cephfs.smithi150.knuivd=up:active} 1 up:standby-replay 1 up:standby 2023-12-16T20:49:32.382 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:32 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi052.bvmosa"}]: dispatch 2023-12-16T20:49:32.382 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:32 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:32.382 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:32 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:32.382 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:32 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:32.382 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:32 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:49:32.382 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:32 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:32.382 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:32 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:49:32.383 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.381+0000 7f6aa67fc700 1 -- 172.21.15.52:0/387112917 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1847 (secure 0 0 0) 0x7f6aac057c00 con 0x7f6ab00fbaf0 2023-12-16T20:49:32.383 INFO:teuthology.orchestra.run.smithi052.stdout:e16 2023-12-16T20:49:32.383 INFO:teuthology.orchestra.run.smithi052.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-12-16T20:49:32.383 INFO:teuthology.orchestra.run.smithi052.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T20:49:32.383 INFO:teuthology.orchestra.run.smithi052.stdout:legacy client fscid: 1 2023-12-16T20:49:32.383 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:49:32.383 INFO:teuthology.orchestra.run.smithi052.stdout:Filesystem 'cephfs' (1) 2023-12-16T20:49:32.383 INFO:teuthology.orchestra.run.smithi052.stdout:fs_name cephfs 2023-12-16T20:49:32.383 INFO:teuthology.orchestra.run.smithi052.stdout:epoch 16 2023-12-16T20:49:32.384 INFO:teuthology.orchestra.run.smithi052.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-12-16T20:49:32.384 INFO:teuthology.orchestra.run.smithi052.stdout:created 2023-12-16T20:49:20.450316+0000 2023-12-16T20:49:32.384 INFO:teuthology.orchestra.run.smithi052.stdout:modified 2023-12-16T20:49:32.124816+0000 2023-12-16T20:49:32.384 INFO:teuthology.orchestra.run.smithi052.stdout:tableserver 0 2023-12-16T20:49:32.384 INFO:teuthology.orchestra.run.smithi052.stdout:root 0 2023-12-16T20:49:32.384 INFO:teuthology.orchestra.run.smithi052.stdout:session_timeout 60 2023-12-16T20:49:32.384 INFO:teuthology.orchestra.run.smithi052.stdout:session_autoclose 300 2023-12-16T20:49:32.384 INFO:teuthology.orchestra.run.smithi052.stdout:max_file_size 1099511627776 2023-12-16T20:49:32.384 INFO:teuthology.orchestra.run.smithi052.stdout:required_client_features {} 2023-12-16T20:49:32.384 INFO:teuthology.orchestra.run.smithi052.stdout:last_failure 0 2023-12-16T20:49:32.384 INFO:teuthology.orchestra.run.smithi052.stdout:last_failure_osd_epoch 43 2023-12-16T20:49:32.384 INFO:teuthology.orchestra.run.smithi052.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T20:49:32.385 INFO:teuthology.orchestra.run.smithi052.stdout:max_mds 2 2023-12-16T20:49:32.385 INFO:teuthology.orchestra.run.smithi052.stdout:in 0,1 2023-12-16T20:49:32.385 INFO:teuthology.orchestra.run.smithi052.stdout:up {0=14486,1=24263} 2023-12-16T20:49:32.385 INFO:teuthology.orchestra.run.smithi052.stdout:failed 2023-12-16T20:49:32.385 INFO:teuthology.orchestra.run.smithi052.stdout:damaged 2023-12-16T20:49:32.385 INFO:teuthology.orchestra.run.smithi052.stdout:stopped 2023-12-16T20:49:32.385 INFO:teuthology.orchestra.run.smithi052.stdout:data_pools [3] 2023-12-16T20:49:32.385 INFO:teuthology.orchestra.run.smithi052.stdout:metadata_pool 2 2023-12-16T20:49:32.385 INFO:teuthology.orchestra.run.smithi052.stdout:inline_data enabled 2023-12-16T20:49:32.386 INFO:teuthology.orchestra.run.smithi052.stdout:balancer 2023-12-16T20:49:32.386 INFO:teuthology.orchestra.run.smithi052.stdout:standby_count_wanted 1 2023-12-16T20:49:32.386 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.cephfs.smithi052.szlsle{0:14486} state up:reconnect seq 4 join_fscid=1 addr [v2:172.21.15.52:6828/1781554610,v1:172.21.15.52:6829/1781554610] compat {c=[1],r=[1],i=[7ff]}] 2023-12-16T20:49:32.386 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.cephfs.smithi150.knuivd{1:24263} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.150:6824/351844450,v1:172.21.15.150:6825/351844450] compat {c=[1],r=[1],i=[7ff]}] 2023-12-16T20:49:32.386 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.cephfs.smithi150.njrgli{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.150:6826/4088837116,v1:172.21.15.150:6827/4088837116] compat {c=[1],r=[1],i=[7ff]}] 2023-12-16T20:49:32.386 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:49:32.386 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:49:32.386 INFO:teuthology.orchestra.run.smithi052.stdout:Standby daemons: 2023-12-16T20:49:32.386 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:49:32.386 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.cephfs.smithi052.bvmosa{-1:24287} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.52:6826/2170358701,v1:172.21.15.52:6827/2170358701] compat {c=[1],r=[1],i=[7ff]}] 2023-12-16T20:49:32.387 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.383+0000 7f6ab7ad0700 1 -- 172.21.15.52:0/387112917 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6a9c068f70 msgr2=0x7f6a9c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:32.387 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.383+0000 7f6ab7ad0700 1 --2- 172.21.15.52:0/387112917 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6a9c068f70 0x7f6a9c06b420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0009ca0 tx=0x7f6aa0009250 comp rx=0 tx=0).stop 2023-12-16T20:49:32.387 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.383+0000 7f6ab7ad0700 1 -- 172.21.15.52:0/387112917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab00fbaf0 msgr2=0x7f6ab010f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:32.387 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.383+0000 7f6ab7ad0700 1 --2- 172.21.15.52:0/387112917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab00fbaf0 0x7f6ab010f770 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6aac009fd0 tx=0x7f6aac009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:32.387 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.384+0000 7f6ab7ad0700 1 -- 172.21.15.52:0/387112917 shutdown_connections 2023-12-16T20:49:32.387 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.384+0000 7f6ab7ad0700 1 --2- 172.21.15.52:0/387112917 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6a9c068f70 0x7f6a9c06b420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:32.387 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.384+0000 7f6ab7ad0700 1 --2- 172.21.15.52:0/387112917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab00fbaf0 0x7f6ab010f770 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:32.387 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.384+0000 7f6ab7ad0700 1 --2- 172.21.15.52:0/387112917 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ab00fcd00 0x7f6ab010fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:32.387 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.384+0000 7f6ab7ad0700 1 -- 172.21.15.52:0/387112917 >> 172.21.15.52:0/387112917 conn(0x7f6ab00f71d0 msgr2=0x7f6ab0065c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:32.388 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.384+0000 7f6ab7ad0700 1 -- 172.21.15.52:0/387112917 shutdown_connections 2023-12-16T20:49:32.388 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:32.384+0000 7f6ab7ad0700 1 -- 172.21.15.52:0/387112917 wait complete. 2023-12-16T20:49:32.388 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 16 2023-12-16T20:49:32.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:32 smithi150 ceph-mon[88000]: mds.? [v2:172.21.15.52:6826/2170358701,v1:172.21.15.52:6827/2170358701] up:boot 2023-12-16T20:49:32.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:32 smithi150 ceph-mon[88000]: mds.? [v2:172.21.15.52:6828/1781554610,v1:172.21.15.52:6829/1781554610] up:resolve 2023-12-16T20:49:32.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:32 smithi150 ceph-mon[88000]: fsmap cephfs:2/2 {0=cephfs.smithi052.szlsle=up:resolve,1=cephfs.smithi150.knuivd=up:active} 1 up:standby-replay 1 up:standby 2023-12-16T20:49:32.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi052.bvmosa"}]: dispatch 2023-12-16T20:49:32.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:32.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:32.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:32.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:49:32.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:32.439 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:49:32.736 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2023-12-16T20:49:33.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:33 smithi150 ceph-mon[88000]: pgmap v128: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2023-12-16T20:49:33.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:33 smithi150 ceph-mon[88000]: mds.? [v2:172.21.15.52:6828/1781554610,v1:172.21.15.52:6829/1781554610] up:reconnect 2023-12-16T20:49:33.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:33 smithi150 ceph-mon[88000]: mds.? [v2:172.21.15.150:6826/4088837116,v1:172.21.15.150:6827/4088837116] up:standby-replay 2023-12-16T20:49:33.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:33 smithi150 ceph-mon[88000]: fsmap cephfs:2/2 {0=cephfs.smithi052.szlsle=up:reconnect,1=cephfs.smithi150.knuivd=up:active} 1 up:standby-replay 1 up:standby 2023-12-16T20:49:33.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:33 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/387112917' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-16T20:49:33.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:33 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:33.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:33 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:49:33.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:33 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:33.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:33 smithi052 ceph-mon[80761]: pgmap v128: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2023-12-16T20:49:33.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:33 smithi052 ceph-mon[80761]: mds.? [v2:172.21.15.52:6828/1781554610,v1:172.21.15.52:6829/1781554610] up:reconnect 2023-12-16T20:49:33.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:33 smithi052 ceph-mon[80761]: mds.? [v2:172.21.15.150:6826/4088837116,v1:172.21.15.150:6827/4088837116] up:standby-replay 2023-12-16T20:49:33.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:33 smithi052 ceph-mon[80761]: fsmap cephfs:2/2 {0=cephfs.smithi052.szlsle=up:reconnect,1=cephfs.smithi150.knuivd=up:active} 1 up:standby-replay 1 up:standby 2023-12-16T20:49:33.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:33 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/387112917' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-16T20:49:33.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:33 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:33.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:33 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:49:33.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:33 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:33.306 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:34.411 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:34 smithi052 ceph-mon[80761]: mds.? [v2:172.21.15.52:6828/1781554610,v1:172.21.15.52:6829/1781554610] up:rejoin 2023-12-16T20:49:34.411 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:34 smithi052 ceph-mon[80761]: fsmap cephfs:2/2 {0=cephfs.smithi052.szlsle=up:rejoin,1=cephfs.smithi150.knuivd=up:active} 1 up:standby-replay 1 up:standby 2023-12-16T20:49:34.411 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:34 smithi052 ceph-mon[80761]: daemon mds.cephfs.smithi052.szlsle is now active in filesystem cephfs as rank 0 2023-12-16T20:49:34.411 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:34 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:34.411 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:34 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:34.411 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:34 smithi052 ceph-mon[80761]: pgmap v129: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2023-12-16T20:49:34.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:34 smithi150 ceph-mon[88000]: mds.? [v2:172.21.15.52:6828/1781554610,v1:172.21.15.52:6829/1781554610] up:rejoin 2023-12-16T20:49:34.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:34 smithi150 ceph-mon[88000]: fsmap cephfs:2/2 {0=cephfs.smithi052.szlsle=up:rejoin,1=cephfs.smithi150.knuivd=up:active} 1 up:standby-replay 1 up:standby 2023-12-16T20:49:34.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:34 smithi150 ceph-mon[88000]: daemon mds.cephfs.smithi052.szlsle is now active in filesystem cephfs as rank 0 2023-12-16T20:49:34.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:34 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:34.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:34 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:34.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:34 smithi150 ceph-mon[88000]: pgmap v129: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2023-12-16T20:49:34.516 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.515+0000 7feb9cf5e700 1 -- 172.21.15.52:0/2457292815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb980fcbf0 msgr2=0x7feb980fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:34.516 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.515+0000 7feb9cf5e700 1 --2- 172.21.15.52:0/2457292815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb980fcbf0 0x7feb980fd000 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feb88004940 tx=0x7feb88009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:34.516 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.515+0000 7feb9cf5e700 1 -- 172.21.15.52:0/2457292815 shutdown_connections 2023-12-16T20:49:34.516 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.515+0000 7feb9cf5e700 1 --2- 172.21.15.52:0/2457292815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb980fde00 0x7feb980fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:34.516 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.515+0000 7feb9cf5e700 1 --2- 172.21.15.52:0/2457292815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb980fcbf0 0x7feb980fd000 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:34.517 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.515+0000 7feb9cf5e700 1 -- 172.21.15.52:0/2457292815 >> 172.21.15.52:0/2457292815 conn(0x7feb980f82d0 msgr2=0x7feb980fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:34.517 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.515+0000 7feb9cf5e700 1 -- 172.21.15.52:0/2457292815 shutdown_connections 2023-12-16T20:49:34.517 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.515+0000 7feb9cf5e700 1 -- 172.21.15.52:0/2457292815 wait complete. 2023-12-16T20:49:34.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.516+0000 7feb9cf5e700 1 Processor -- start 2023-12-16T20:49:34.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.516+0000 7feb9cf5e700 1 -- start start 2023-12-16T20:49:34.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.516+0000 7feb9cf5e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb980fcbf0 0x7feb9810d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:34.518 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.516+0000 7feb9cf5e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb980fde00 0x7feb9810daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:34.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.516+0000 7feb9cf5e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9810e0a0 con 0x7feb980fcbf0 2023-12-16T20:49:34.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.516+0000 7feb9cf5e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9810e940 con 0x7feb980fde00 2023-12-16T20:49:34.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.516+0000 7feb9659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb980fcbf0 0x7feb9810d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:34.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.516+0000 7feb9659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb980fcbf0 0x7feb9810d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34420/0 (socket says 172.21.15.52:34420) 2023-12-16T20:49:34.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.516+0000 7feb9659c700 1 -- 172.21.15.52:0/3590556688 learned_addr learned my addr 172.21.15.52:0/3590556688 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:34.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.517+0000 7feb95d9b700 1 --2- 172.21.15.52:0/3590556688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb980fde00 0x7feb9810daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:34.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.517+0000 7feb9659c700 1 -- 172.21.15.52:0/3590556688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb980fde00 msgr2=0x7feb9810daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:34.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.517+0000 7feb9659c700 1 --2- 172.21.15.52:0/3590556688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb980fde00 0x7feb9810daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:34.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.517+0000 7feb9659c700 1 -- 172.21.15.52:0/3590556688 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb88005040 con 0x7feb980fcbf0 2023-12-16T20:49:34.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.517+0000 7feb95d9b700 1 --2- 172.21.15.52:0/3590556688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb980fde00 0x7feb9810daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T20:49:34.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.517+0000 7feb9659c700 1 --2- 172.21.15.52:0/3590556688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb980fcbf0 0x7feb9810d560 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7feb88004f90 tx=0x7feb88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:34.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.517+0000 7feb86ffd700 1 -- 172.21.15.52:0/3590556688 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb8800e040 con 0x7feb980fcbf0 2023-12-16T20:49:34.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.517+0000 7feb9cf5e700 1 -- 172.21.15.52:0/3590556688 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9810ebc0 con 0x7feb980fcbf0 2023-12-16T20:49:34.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.517+0000 7feb86ffd700 1 -- 172.21.15.52:0/3590556688 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb8800bcd0 con 0x7feb980fcbf0 2023-12-16T20:49:34.519 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.517+0000 7feb86ffd700 1 -- 172.21.15.52:0/3590556688 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb8801c630 con 0x7feb980fcbf0 2023-12-16T20:49:34.520 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.517+0000 7feb9cf5e700 1 -- 172.21.15.52:0/3590556688 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb9810f100 con 0x7feb980fcbf0 2023-12-16T20:49:34.520 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.518+0000 7feb9cf5e700 1 -- 172.21.15.52:0/3590556688 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb981075d0 con 0x7feb980fcbf0 2023-12-16T20:49:34.520 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.519+0000 7feb86ffd700 1 -- 172.21.15.52:0/3590556688 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7feb88013960 con 0x7feb980fcbf0 2023-12-16T20:49:34.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.520+0000 7feb86ffd700 1 --2- 172.21.15.52:0/3590556688 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7feb80068f60 0x7feb8006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:34.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.520+0000 7feb86ffd700 1 -- 172.21.15.52:0/3590556688 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7feb880881b0 con 0x7feb980fcbf0 2023-12-16T20:49:34.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.520+0000 7feb95d9b700 1 --2- 172.21.15.52:0/3590556688 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7feb80068f60 0x7feb8006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:34.521 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.520+0000 7feb95d9b700 1 --2- 172.21.15.52:0/3590556688 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7feb80068f60 0x7feb8006b410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009fd0 tx=0x7feb8c009ee0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:34.525 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.524+0000 7feb86ffd700 1 -- 172.21.15.52:0/3590556688 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7feb88058160 con 0x7feb980fcbf0 2023-12-16T20:49:34.769 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.768+0000 7feb9cf5e700 1 -- 172.21.15.52:0/3590556688 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb9804bb80 con 0x7feb980fcbf0 2023-12-16T20:49:34.770 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.769+0000 7feb86ffd700 1 -- 172.21.15.52:0/3590556688 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4722 (secure 0 0 0) 0x7feb88057cf0 con 0x7feb980fcbf0 2023-12-16T20:49:34.772 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.771+0000 7feb9cf5e700 1 -- 172.21.15.52:0/3590556688 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7feb80068f60 msgr2=0x7feb8006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:34.772 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.771+0000 7feb9cf5e700 1 --2- 172.21.15.52:0/3590556688 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7feb80068f60 0x7feb8006b410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009fd0 tx=0x7feb8c009ee0 comp rx=0 tx=0).stop 2023-12-16T20:49:34.772 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.771+0000 7feb9cf5e700 1 -- 172.21.15.52:0/3590556688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb980fcbf0 msgr2=0x7feb9810d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:34.772 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.771+0000 7feb9cf5e700 1 --2- 172.21.15.52:0/3590556688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb980fcbf0 0x7feb9810d560 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7feb88004f90 tx=0x7feb88009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:34.772 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.771+0000 7feb9cf5e700 1 -- 172.21.15.52:0/3590556688 shutdown_connections 2023-12-16T20:49:34.772 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.771+0000 7feb9cf5e700 1 --2- 172.21.15.52:0/3590556688 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7feb80068f60 0x7feb8006b410 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:34.772 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.771+0000 7feb9cf5e700 1 --2- 172.21.15.52:0/3590556688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb980fcbf0 0x7feb9810d560 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:34.772 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.771+0000 7feb9cf5e700 1 --2- 172.21.15.52:0/3590556688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb980fde00 0x7feb9810daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:34.772 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.771+0000 7feb9cf5e700 1 -- 172.21.15.52:0/3590556688 >> 172.21.15.52:0/3590556688 conn(0x7feb980f82d0 msgr2=0x7feb980fa590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:34.772 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.771+0000 7feb9cf5e700 1 -- 172.21.15.52:0/3590556688 shutdown_connections 2023-12-16T20:49:34.772 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:34.771+0000 7feb9cf5e700 1 -- 172.21.15.52:0/3590556688 wait complete. 2023-12-16T20:49:34.773 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 19 2023-12-16T20:49:34.782 INFO:teuthology.orchestra.run.smithi052.stdout:true 2023-12-16T20:49:35.177 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2023-12-16T20:49:35.207 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:35 smithi052 ceph-mon[80761]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-12-16T20:49:35.207 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:35 smithi052 ceph-mon[80761]: mds.? [v2:172.21.15.52:6828/1781554610,v1:172.21.15.52:6829/1781554610] up:active 2023-12-16T20:49:35.207 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:35 smithi052 ceph-mon[80761]: fsmap cephfs:2 {0=cephfs.smithi052.szlsle=up:active,1=cephfs.smithi150.knuivd=up:active} 1 up:standby-replay 1 up:standby 2023-12-16T20:49:35.207 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:35 smithi052 ceph-mon[80761]: fsmap cephfs:2 {0=cephfs.smithi052.szlsle=up:active,1=cephfs.smithi150.knuivd=up:active} 2 up:standby-replay 2023-12-16T20:49:35.208 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:35 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3590556688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-16T20:49:35.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:35 smithi150 ceph-mon[88000]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-12-16T20:49:35.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:35 smithi150 ceph-mon[88000]: mds.? [v2:172.21.15.52:6828/1781554610,v1:172.21.15.52:6829/1781554610] up:active 2023-12-16T20:49:35.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:35 smithi150 ceph-mon[88000]: fsmap cephfs:2 {0=cephfs.smithi052.szlsle=up:active,1=cephfs.smithi150.knuivd=up:active} 1 up:standby-replay 1 up:standby 2023-12-16T20:49:35.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:35 smithi150 ceph-mon[88000]: fsmap cephfs:2 {0=cephfs.smithi052.szlsle=up:active,1=cephfs.smithi150.knuivd=up:active} 2 up:standby-replay 2023-12-16T20:49:35.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:35 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/3590556688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-16T20:49:35.558 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:36.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:36 smithi150 ceph-mon[88000]: pgmap v130: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2023-12-16T20:49:36.468 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:36 smithi052 ceph-mon[80761]: pgmap v130: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2023-12-16T20:49:36.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.854+0000 7fc5b7bd5700 1 -- 172.21.15.52:0/518126836 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5b00fc450 msgr2=0x7fc5b00fc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:36.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.854+0000 7fc5b7bd5700 1 --2- 172.21.15.52:0/518126836 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5b00fc450 0x7fc5b00fc8c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0004a50 tx=0x7fc5a000af40 comp rx=0 tx=0).stop 2023-12-16T20:49:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.854+0000 7fc5b7bd5700 1 -- 172.21.15.52:0/518126836 shutdown_connections 2023-12-16T20:49:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.854+0000 7fc5b7bd5700 1 --2- 172.21.15.52:0/518126836 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5b00fc450 0x7fc5b00fc8c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.854+0000 7fc5b7bd5700 1 --2- 172.21.15.52:0/518126836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b00fb590 0x7fc5b00fb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.854+0000 7fc5b7bd5700 1 -- 172.21.15.52:0/518126836 >> 172.21.15.52:0/518126836 conn(0x7fc5b00f64f0 msgr2=0x7fc5b00f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.854+0000 7fc5b7bd5700 1 -- 172.21.15.52:0/518126836 shutdown_connections 2023-12-16T20:49:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.854+0000 7fc5b7bd5700 1 -- 172.21.15.52:0/518126836 wait complete. 2023-12-16T20:49:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.855+0000 7fc5b7bd5700 1 Processor -- start 2023-12-16T20:49:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.855+0000 7fc5b7bd5700 1 -- start start 2023-12-16T20:49:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.855+0000 7fc5b7bd5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5b00fb590 0x7fc5b010d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.855+0000 7fc5b7bd5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b00fc450 0x7fc5b010d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.855+0000 7fc5b7bd5700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b010dff0 con 0x7fc5b00fb590 2023-12-16T20:49:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.855+0000 7fc5b7bd5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b010e890 con 0x7fc5b00fc450 2023-12-16T20:49:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.855+0000 7fc5b5971700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5b00fb590 0x7fc5b010d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:36.860 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.855+0000 7fc5b5971700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5b00fb590 0x7fc5b010d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34444/0 (socket says 172.21.15.52:34444) 2023-12-16T20:49:36.860 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.855+0000 7fc5b5971700 1 -- 172.21.15.52:0/2309933344 learned_addr learned my addr 172.21.15.52:0/2309933344 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:36.860 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.855+0000 7fc5b5170700 1 --2- 172.21.15.52:0/2309933344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b00fc450 0x7fc5b010d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:36.860 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.855+0000 7fc5b5971700 1 -- 172.21.15.52:0/2309933344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b00fc450 msgr2=0x7fc5b010d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:36.860 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.855+0000 7fc5b5971700 1 --2- 172.21.15.52:0/2309933344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b00fc450 0x7fc5b010d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:36.860 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.855+0000 7fc5b5971700 1 -- 172.21.15.52:0/2309933344 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a0005040 con 0x7fc5b00fb590 2023-12-16T20:49:36.860 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.855+0000 7fc5b5971700 1 --2- 172.21.15.52:0/2309933344 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5b00fb590 0x7fc5b010d4b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac002a80 tx=0x7fc5ac004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:36.860 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.855+0000 7fc5a67fc700 1 -- 172.21.15.52:0/2309933344 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5ac007cc0 con 0x7fc5b00fb590 2023-12-16T20:49:36.861 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.856+0000 7fc5a67fc700 1 -- 172.21.15.52:0/2309933344 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5ac00f040 con 0x7fc5b00fb590 2023-12-16T20:49:36.861 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.856+0000 7fc5b7bd5700 1 -- 172.21.15.52:0/2309933344 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b010eb70 con 0x7fc5b00fb590 2023-12-16T20:49:36.861 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.856+0000 7fc5a67fc700 1 -- 172.21.15.52:0/2309933344 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5ac00d9c0 con 0x7fc5b00fb590 2023-12-16T20:49:36.861 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.856+0000 7fc5b7bd5700 1 -- 172.21.15.52:0/2309933344 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b010f0b0 con 0x7fc5b00fb590 2023-12-16T20:49:36.861 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.857+0000 7fc5b7bd5700 1 -- 172.21.15.52:0/2309933344 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b01073f0 con 0x7fc5b00fb590 2023-12-16T20:49:36.863 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.857+0000 7fc5a67fc700 1 -- 172.21.15.52:0/2309933344 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fc5ac00db20 con 0x7fc5b00fb590 2023-12-16T20:49:36.863 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.857+0000 7fc5a67fc700 1 --2- 172.21.15.52:0/2309933344 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc59c068f60 0x7fc59c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:36.863 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.857+0000 7fc5a67fc700 1 -- 172.21.15.52:0/2309933344 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc5ac086b10 con 0x7fc5b00fb590 2023-12-16T20:49:36.863 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.857+0000 7fc5b5170700 1 --2- 172.21.15.52:0/2309933344 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc59c068f60 0x7fc59c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:36.863 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.858+0000 7fc5b5170700 1 --2- 172.21.15.52:0/2309933344 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc59c068f60 0x7fc59c06b410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000a5b0 tx=0x7fc5a000af40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:36.863 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:36.862+0000 7fc5a67fc700 1 -- 172.21.15.52:0/2309933344 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc5ac0569a0 con 0x7fc5b00fb590 2023-12-16T20:49:37.150 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.150+0000 7fc5b7bd5700 1 -- 172.21.15.52:0/2309933344 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fc5b004bb80 con 0x7fc5b00fb590 2023-12-16T20:49:37.151 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.150+0000 7fc5a67fc700 1 -- 172.21.15.52:0/2309933344 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v19) v1 ==== 78+0+99 (secure 0 0 0) 0x7fc5ac056530 con 0x7fc5b00fb590 2023-12-16T20:49:37.153 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.153+0000 7fc5b7bd5700 1 -- 172.21.15.52:0/2309933344 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc59c068f60 msgr2=0x7fc59c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:37.154 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.153+0000 7fc5b7bd5700 1 --2- 172.21.15.52:0/2309933344 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc59c068f60 0x7fc59c06b410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000a5b0 tx=0x7fc5a000af40 comp rx=0 tx=0).stop 2023-12-16T20:49:37.154 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.153+0000 7fc5b7bd5700 1 -- 172.21.15.52:0/2309933344 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5b00fb590 msgr2=0x7fc5b010d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:37.154 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.153+0000 7fc5b7bd5700 1 --2- 172.21.15.52:0/2309933344 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5b00fb590 0x7fc5b010d4b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac002a80 tx=0x7fc5ac004600 comp rx=0 tx=0).stop 2023-12-16T20:49:37.154 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.153+0000 7fc5b7bd5700 1 -- 172.21.15.52:0/2309933344 shutdown_connections 2023-12-16T20:49:37.154 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.153+0000 7fc5b7bd5700 1 --2- 172.21.15.52:0/2309933344 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fc59c068f60 0x7fc59c06b410 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:37.154 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.153+0000 7fc5b7bd5700 1 --2- 172.21.15.52:0/2309933344 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5b00fb590 0x7fc5b010d4b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:37.154 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.153+0000 7fc5b7bd5700 1 --2- 172.21.15.52:0/2309933344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b00fc450 0x7fc5b010d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:37.154 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.153+0000 7fc5b7bd5700 1 -- 172.21.15.52:0/2309933344 >> 172.21.15.52:0/2309933344 conn(0x7fc5b00f64f0 msgr2=0x7fc5b00ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:37.154 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.153+0000 7fc5b7bd5700 1 -- 172.21.15.52:0/2309933344 shutdown_connections 2023-12-16T20:49:37.154 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.153+0000 7fc5b7bd5700 1 -- 172.21.15.52:0/2309933344 wait complete. 2023-12-16T20:49:37.165 INFO:teuthology.orchestra.run.smithi052.stdout:true 2023-12-16T20:49:37.493 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:37 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/2309933344' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2023-12-16T20:49:37.520 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2023-12-16T20:49:37.533 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T20:49:37.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:37 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/2309933344' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2023-12-16T20:49:37.690 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.689+0000 7f801b8ae640 1 -- 172.21.15.52:0/1925316791 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f80140fc060 msgr2=0x7f80140fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:37.690 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.689+0000 7f801b8ae640 1 --2- 172.21.15.52:0/1925316791 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f80140fc060 0x7f80140fc4d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f80080048d0 tx=0x7f8008009da0 comp rx=0 tx=0).stop 2023-12-16T20:49:37.690 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.689+0000 7f801b8ae640 1 -- 172.21.15.52:0/1925316791 shutdown_connections 2023-12-16T20:49:37.690 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.689+0000 7f801b8ae640 1 --2- 172.21.15.52:0/1925316791 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f80140fc060 0x7f80140fc4d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:37.690 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.689+0000 7f801b8ae640 1 --2- 172.21.15.52:0/1925316791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80140faed0 0x7f80140fb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:37.690 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.689+0000 7f801b8ae640 1 -- 172.21.15.52:0/1925316791 >> 172.21.15.52:0/1925316791 conn(0x7f80140f65f0 msgr2=0x7f80140f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:37.690 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.689+0000 7f801b8ae640 1 -- 172.21.15.52:0/1925316791 shutdown_connections 2023-12-16T20:49:37.691 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.689+0000 7f801b8ae640 1 -- 172.21.15.52:0/1925316791 wait complete. 2023-12-16T20:49:37.691 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.690+0000 7f801b8ae640 1 Processor -- start 2023-12-16T20:49:37.691 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.690+0000 7f801b8ae640 1 -- start start 2023-12-16T20:49:37.692 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.690+0000 7f801b8ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80140faed0 0x7f801410b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:37.692 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.690+0000 7f801b8ae640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f80140fc060 0x7f801410bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:37.693 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.690+0000 7f801b8ae640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801410c1c0 con 0x7f80140fc060 2023-12-16T20:49:37.693 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.690+0000 7f801b8ae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8014110c10 con 0x7f80140faed0 2023-12-16T20:49:37.693 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.690+0000 7f8018e49640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f80140fc060 0x7f801410bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:37.693 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.690+0000 7f8018e49640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f80140fc060 0x7f801410bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34458/0 (socket says 172.21.15.52:34458) 2023-12-16T20:49:37.693 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.690+0000 7f8018e49640 1 -- 172.21.15.52:0/3342902795 learned_addr learned my addr 172.21.15.52:0/3342902795 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:37.693 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.690+0000 7f801964a640 1 --2- 172.21.15.52:0/3342902795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80140faed0 0x7f801410b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:37.693 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.690+0000 7f8018e49640 1 -- 172.21.15.52:0/3342902795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80140faed0 msgr2=0x7f801410b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:37.693 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.690+0000 7f8018e49640 1 --2- 172.21.15.52:0/3342902795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80140faed0 0x7f801410b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:37.693 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.690+0000 7f8018e49640 1 -- 172.21.15.52:0/3342902795 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8008005040 con 0x7f80140fc060 2023-12-16T20:49:37.693 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.690+0000 7f801964a640 1 --2- 172.21.15.52:0/3342902795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80140faed0 0x7f801410b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T20:49:37.693 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.690+0000 7f8018e49640 1 --2- 172.21.15.52:0/3342902795 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f80140fc060 0x7f801410bb70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8008007da0 tx=0x7f8008009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:37.693 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.691+0000 7f8011ffb640 1 -- 172.21.15.52:0/3342902795 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80080094c0 con 0x7f80140fc060 2023-12-16T20:49:37.693 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.691+0000 7f801b8ae640 1 -- 172.21.15.52:0/3342902795 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8014110d50 con 0x7f80140fc060 2023-12-16T20:49:37.694 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.691+0000 7f8011ffb640 1 -- 172.21.15.52:0/3342902795 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8008009620 con 0x7f80140fc060 2023-12-16T20:49:37.694 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.691+0000 7f8011ffb640 1 -- 172.21.15.52:0/3342902795 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f800803b400 con 0x7f80140fc060 2023-12-16T20:49:37.694 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.691+0000 7f801b8ae640 1 -- 172.21.15.52:0/3342902795 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80141112c0 con 0x7f80140fc060 2023-12-16T20:49:37.694 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.692+0000 7f801b8ae640 1 -- 172.21.15.52:0/3342902795 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8014103860 con 0x7f80140fc060 2023-12-16T20:49:37.694 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.693+0000 7f8011ffb640 1 -- 172.21.15.52:0/3342902795 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f800802f070 con 0x7f80140fc060 2023-12-16T20:49:37.695 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.693+0000 7f8011ffb640 1 --2- 172.21.15.52:0/3342902795 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7ffc068ed0 0x7f7ffc06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:37.695 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.693+0000 7f8011ffb640 1 -- 172.21.15.52:0/3342902795 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f80080a9380 con 0x7f80140fc060 2023-12-16T20:49:37.695 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.693+0000 7f801964a640 1 --2- 172.21.15.52:0/3342902795 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7ffc068ed0 0x7f7ffc06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:37.695 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.693+0000 7f801964a640 1 --2- 172.21.15.52:0/3342902795 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7ffc068ed0 0x7f7ffc06b380 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f800c0049d0 tx=0x7f800c00b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:37.698 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.698+0000 7f8011ffb640 1 -- 172.21.15.52:0/3342902795 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8008075780 con 0x7f80140fc060 2023-12-16T20:49:37.873 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.872+0000 7f801b8ae640 1 -- 172.21.15.52:0/3342902795 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8014111570 con 0x7f80140fc060 2023-12-16T20:49:37.878 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.873+0000 7f8011ffb640 1 -- 172.21.15.52:0/3342902795 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+14393 (secure 0 0 0) 0x7f8008078da0 con 0x7f80140fc060 2023-12-16T20:49:37.879 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:49:37.879 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":43,"fsid":"cf043a8a-9c53-11ee-95a5-87774f69a715","created":"2023-12-16T20:44:44.679887+0000","modified":"2023-12-16T20:49:30.121534+0000","last_up_change":"2023-12-16T20:48:33.120930+0000","last_in_change":"2023-12-16T20:48:17.606842+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T20:47:46.398235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2023-12-16T20:49:19.016135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2023-12-16T20:49:19.883810+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03268674-f4fa-4c69-9fa5-ac2327873b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6803","nonce":362877775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6805","nonce":362877775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6809","nonce":362877775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":362877775},{"type":"v1","addr":"172.21.15.52:6807","nonce":362877775}]},"public_addr":"172.21.15.52:6803/362877775","cluster_addr":"172.21.15.52:6805/362877775","heartbeat_back_addr":"172.21.15.52:6809/362877775","heartbeat_front_addr":"172.21.15.52:6807/362877775","state":["exists","up"]},{"osd":1,"uuid":"8cf65e84-7be8-4d6c-872d-d908e7a55c46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6811","nonce":2532880721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6813","nonce":2532880721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6817","nonce":2532880721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":2532880721},{"type":"v1","addr":"172.21.15.52:6815","nonce":2532880721}]},"public_addr":"172.21.15.52:6811/2532880721","cluster_addr":"172.21.15.52:6813/2532880721","heartbeat_back_addr":"172.21.15.52:6817/2532880721","heartbeat_front_addr":"172.21.15.52:6815/2532880721","state":["exists","up"]},{"osd":2,"uuid":"fd375c62-57d0-4899-aa93-b582a664abbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6819","nonce":4281298930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6821","nonce":4281298930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6825","nonce":4281298930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4281298930},{"type":"v1","addr":"172.21.15.52:6823","nonce":4281298930}]},"public_addr":"172.21.15.52:6819/4281298930","cluster_addr":"172.21.15.52:6821/4281298930","heartbeat_back_addr":"172.21.15.52:6825/4281298930","heartbeat_front_addr":"172.21.15.52:6823/4281298930","state":["exists","up"]},{"osd":3,"uuid":"31298367-7548-4ef5-bd2b-a6f586ec14a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6801","nonce":2861125970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6803","nonce":2861125970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6807","nonce":2861125970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":2861125970},{"type":"v1","addr":"172.21.15.150:6805","nonce":2861125970}]},"public_addr":"172.21.15.150:6801/2861125970","cluster_addr":"172.21.15.150:6803/2861125970","heartbeat_back_addr":"172.21.15.150:6807/2861125970","heartbeat_front_addr":"172.21.15.150:6805/2861125970","state":["exists","up"]},{"osd":4,"uuid":"ab02c027-f21c-4187-af78-f4daaa844db1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6809","nonce":2997946868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6811","nonce":2997946868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6815","nonce":2997946868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2997946868},{"type":"v1","addr":"172.21.15.150:6813","nonce":2997946868}]},"public_addr":"172.21.15.150:6809/2997946868","cluster_addr":"172.21.15.150:6811/2997946868","heartbeat_back_addr":"172.21.15.150:6815/2997946868","heartbeat_front_addr":"172.21.15.150:6813/2997946868","state":["exists","up"]},{"osd":5,"uuid":"f62123bf-fcd1-4253-b8c7-78c2379bb90b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6817","nonce":4046379778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6819","nonce":4046379778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6823","nonce":4046379778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4046379778},{"type":"v1","addr":"172.21.15.150:6821","nonce":4046379778}]},"public_addr":"172.21.15.150:6817/4046379778","cluster_addr":"172.21.15.150:6819/4046379778","heartbeat_back_addr":"172.21.15.150:6823/4046379778","heartbeat_front_addr":"172.21.15.150:6821/4046379778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:06.709733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:25.479953+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:44.608913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:47:59.806350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:48:15.471651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T20:48:31.147029+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6827/1882583859":"2023-12-17T20:49:30.121510+0000","172.21.15.52:0/27878571":"2023-12-17T20:45:07.199601+0000","172.21.15.52:6801/2740978054":"2023-12-17T20:45:07.199601+0000","172.21.15.52:6801/221719009":"2023-12-17T20:45:28.570795+0000","172.21.15.52:0/1900058090":"2023-12-17T20:46:17.883785+0000","172.21.15.52:6800/2740978054":"2023-12-17T20:45:07.199601+0000","172.21.15.52:0/1810720520":"2023-12-17T20:45:28.570795+0000","172.21.15.52:0/82552375":"2023-12-17T20:45:07.199601+0000","172.21.15.52:0/806767676":"2023-12-17T20:45:07.199601+0000","172.21.15.52:0/2975128681":"2023-12-17T20:45:28.570795+0000","172.21.15.52:6801/2799692744":"2023-12-17T20:46:17.883785+0000","172.21.15.52:0/2730302642":"2023-12-17T20:45:28.570795+0000","172.21.15.52:6800/221719009":"2023-12-17T20:45:28.570795+0000","172.21.15.52:0/2518812785":"2023-12-17T20:46:17.883785+0000","172.21.15.52:0/4160218759":"2023-12-17T20:46:17.883785+0000","172.21.15.52:6800/2799692744":"2023-12-17T20:46:17.883785+0000","172.21.15.52:6826/1882583859":"2023-12-17T20:49:30.121510+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T20:49:37.880 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.875+0000 7f801b8ae640 1 -- 172.21.15.52:0/3342902795 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7ffc068ed0 msgr2=0x7f7ffc06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:37.880 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.875+0000 7f801b8ae640 1 --2- 172.21.15.52:0/3342902795 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7ffc068ed0 0x7f7ffc06b380 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f800c0049d0 tx=0x7f800c00b040 comp rx=0 tx=0).stop 2023-12-16T20:49:37.880 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.875+0000 7f801b8ae640 1 -- 172.21.15.52:0/3342902795 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f80140fc060 msgr2=0x7f801410bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:37.880 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.875+0000 7f801b8ae640 1 --2- 172.21.15.52:0/3342902795 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f80140fc060 0x7f801410bb70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8008007da0 tx=0x7f8008009da0 comp rx=0 tx=0).stop 2023-12-16T20:49:37.880 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.875+0000 7f801b8ae640 1 -- 172.21.15.52:0/3342902795 shutdown_connections 2023-12-16T20:49:37.880 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.875+0000 7f801b8ae640 1 --2- 172.21.15.52:0/3342902795 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7ffc068ed0 0x7f7ffc06b380 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:37.880 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.875+0000 7f801b8ae640 1 --2- 172.21.15.52:0/3342902795 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f80140fc060 0x7f801410bb70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:37.880 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.875+0000 7f801b8ae640 1 --2- 172.21.15.52:0/3342902795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80140faed0 0x7f801410b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:37.880 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.875+0000 7f801b8ae640 1 -- 172.21.15.52:0/3342902795 >> 172.21.15.52:0/3342902795 conn(0x7f80140f65f0 msgr2=0x7f80140ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:37.880 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.875+0000 7f801b8ae640 1 -- 172.21.15.52:0/3342902795 shutdown_connections 2023-12-16T20:49:37.881 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.875+0000 7f801b8ae640 1 -- 172.21.15.52:0/3342902795 wait complete. 2023-12-16T20:49:37.887 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T20:47:46.398235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2023-12-16T20:49:19.016135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2023-12-16T20:49:19.883810+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-16T20:49:37.887 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T20:49:37.995 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.993+0000 7fd8e375d640 1 -- 172.21.15.52:0/1007960916 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8dc0fd310 msgr2=0x7fd8dc0fd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:37.995 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.993+0000 7fd8e375d640 1 --2- 172.21.15.52:0/1007960916 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8dc0fd310 0x7fd8dc0fd780 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc0048d0 tx=0x7fd8cc009da0 comp rx=0 tx=0).stop 2023-12-16T20:49:37.995 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.993+0000 7fd8e375d640 1 -- 172.21.15.52:0/1007960916 shutdown_connections 2023-12-16T20:49:37.995 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.993+0000 7fd8e375d640 1 --2- 172.21.15.52:0/1007960916 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8dc0fd310 0x7fd8dc0fd780 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:37.996 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.993+0000 7fd8e375d640 1 --2- 172.21.15.52:0/1007960916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8dc0fa260 0x7fd8dc0fa670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:37.996 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.993+0000 7fd8e375d640 1 -- 172.21.15.52:0/1007960916 >> 172.21.15.52:0/1007960916 conn(0x7fd8dc0f5d40 msgr2=0x7fd8dc0f8170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:37.996 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.993+0000 7fd8e375d640 1 -- 172.21.15.52:0/1007960916 shutdown_connections 2023-12-16T20:49:37.996 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.993+0000 7fd8e375d640 1 -- 172.21.15.52:0/1007960916 wait complete. 2023-12-16T20:49:37.996 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.993+0000 7fd8e375d640 1 Processor -- start 2023-12-16T20:49:37.996 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.993+0000 7fd8e375d640 1 -- start start 2023-12-16T20:49:37.996 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8e375d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8dc0fa260 0x7fd8dc10c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:37.996 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8e375d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8dc0fd310 0x7fd8dc10cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:37.996 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8e375d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc10d370 con 0x7fd8dc0fa260 2023-12-16T20:49:37.996 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8e375d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc111dc0 con 0x7fd8dc0fd310 2023-12-16T20:49:37.996 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8e14f9640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8dc0fa260 0x7fd8dc10c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:37.996 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8e14f9640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8dc0fa260 0x7fd8dc10c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34480/0 (socket says 172.21.15.52:34480) 2023-12-16T20:49:37.996 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8e14f9640 1 -- 172.21.15.52:0/296626772 learned_addr learned my addr 172.21.15.52:0/296626772 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:37.996 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8e0cf8640 1 --2- 172.21.15.52:0/296626772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8dc0fd310 0x7fd8dc10cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:37.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8e14f9640 1 -- 172.21.15.52:0/296626772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8dc0fd310 msgr2=0x7fd8dc10cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:37.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8e14f9640 1 --2- 172.21.15.52:0/296626772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8dc0fd310 0x7fd8dc10cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:37.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8e14f9640 1 -- 172.21.15.52:0/296626772 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc005040 con 0x7fd8dc0fa260 2023-12-16T20:49:37.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8e14f9640 1 --2- 172.21.15.52:0/296626772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8dc0fa260 0x7fd8dc10c7e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd8d800c7e0 tx=0x7fd8d80043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:37.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8d1ffb640 1 -- 172.21.15.52:0/296626772 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8d800a820 con 0x7fd8dc0fa260 2023-12-16T20:49:37.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8d1ffb640 1 -- 172.21.15.52:0/296626772 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd8d800ae30 con 0x7fd8dc0fa260 2023-12-16T20:49:37.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8e375d640 1 -- 172.21.15.52:0/296626772 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8dc111f60 con 0x7fd8dc0fa260 2023-12-16T20:49:37.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.994+0000 7fd8d1ffb640 1 -- 172.21.15.52:0/296626772 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8d80113a0 con 0x7fd8dc0fa260 2023-12-16T20:49:37.997 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.995+0000 7fd8e375d640 1 -- 172.21.15.52:0/296626772 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8dc112470 con 0x7fd8dc0fa260 2023-12-16T20:49:37.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.995+0000 7fd8e375d640 1 -- 172.21.15.52:0/296626772 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8dc1049c0 con 0x7fd8dc0fa260 2023-12-16T20:49:37.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.996+0000 7fd8d1ffb640 1 -- 172.21.15.52:0/296626772 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fd8d800a980 con 0x7fd8dc0fa260 2023-12-16T20:49:37.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.996+0000 7fd8d1ffb640 1 --2- 172.21.15.52:0/296626772 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd8c8068ed0 0x7fd8c806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:37.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.996+0000 7fd8d1ffb640 1 -- 172.21.15.52:0/296626772 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fd8d8086ef0 con 0x7fd8dc0fa260 2023-12-16T20:49:37.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.996+0000 7fd8e0cf8640 1 --2- 172.21.15.52:0/296626772 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd8c8068ed0 0x7fd8c806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:37.998 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:37.997+0000 7fd8e0cf8640 1 --2- 172.21.15.52:0/296626772 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd8c8068ed0 0x7fd8c806b380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc10dd30 tx=0x7fd8cc009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:38.001 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.001+0000 7fd8d1ffb640 1 -- 172.21.15.52:0/296626772 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd8d8053370 con 0x7fd8dc0fa260 2023-12-16T20:49:38.175 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.174+0000 7fd8e375d640 1 -- 172.21.15.52:0/296626772 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd8dc1129d0 con 0x7fd8dc0fa260 2023-12-16T20:49:38.175 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.174+0000 7fd8d1ffb640 1 -- 172.21.15.52:0/296626772 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd8d8056990 con 0x7fd8dc0fa260 2023-12-16T20:49:38.175 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 1 2023-12-16T20:49:38.177 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.176+0000 7fd8e375d640 1 -- 172.21.15.52:0/296626772 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd8c8068ed0 msgr2=0x7fd8c806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:38.177 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.176+0000 7fd8e375d640 1 --2- 172.21.15.52:0/296626772 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd8c8068ed0 0x7fd8c806b380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc10dd30 tx=0x7fd8cc009da0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.177 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.176+0000 7fd8e375d640 1 -- 172.21.15.52:0/296626772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8dc0fa260 msgr2=0x7fd8dc10c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:38.177 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.176+0000 7fd8e375d640 1 --2- 172.21.15.52:0/296626772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8dc0fa260 0x7fd8dc10c7e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd8d800c7e0 tx=0x7fd8d80043c0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.177 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.176+0000 7fd8e375d640 1 -- 172.21.15.52:0/296626772 shutdown_connections 2023-12-16T20:49:38.177 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.176+0000 7fd8e375d640 1 --2- 172.21.15.52:0/296626772 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fd8c8068ed0 0x7fd8c806b380 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.178 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.176+0000 7fd8e375d640 1 --2- 172.21.15.52:0/296626772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8dc0fd310 0x7fd8dc10cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.178 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.176+0000 7fd8e375d640 1 --2- 172.21.15.52:0/296626772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8dc0fa260 0x7fd8dc10c7e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.178 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.176+0000 7fd8e375d640 1 -- 172.21.15.52:0/296626772 >> 172.21.15.52:0/296626772 conn(0x7fd8dc0f5d40 msgr2=0x7fd8dc100540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:38.178 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.176+0000 7fd8e375d640 1 -- 172.21.15.52:0/296626772 shutdown_connections 2023-12-16T20:49:38.178 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.176+0000 7fd8e375d640 1 -- 172.21.15.52:0/296626772 wait complete. 2023-12-16T20:49:38.188 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2023-12-16T20:49:38.298 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.295+0000 7f85eb55d640 1 -- 172.21.15.52:0/3885266894 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85e40fae30 msgr2=0x7f85e40fb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:38.298 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.295+0000 7f85eb55d640 1 --2- 172.21.15.52:0/3885266894 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85e40fae30 0x7f85e40fb240 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f85dc0048d0 tx=0x7f85dc009da0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.298 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.295+0000 7f85eb55d640 1 -- 172.21.15.52:0/3885266894 shutdown_connections 2023-12-16T20:49:38.298 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.295+0000 7f85eb55d640 1 --2- 172.21.15.52:0/3885266894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85e40fc050 0x7f85e40fc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.298 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.295+0000 7f85eb55d640 1 --2- 172.21.15.52:0/3885266894 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85e40fae30 0x7f85e40fb240 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.298 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.295+0000 7f85eb55d640 1 -- 172.21.15.52:0/3885266894 >> 172.21.15.52:0/3885266894 conn(0x7f85e40f6580 msgr2=0x7f85e40f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:38.298 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.295+0000 7f85eb55d640 1 -- 172.21.15.52:0/3885266894 shutdown_connections 2023-12-16T20:49:38.298 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.295+0000 7f85eb55d640 1 -- 172.21.15.52:0/3885266894 wait complete. 2023-12-16T20:49:38.298 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.296+0000 7f85eb55d640 1 Processor -- start 2023-12-16T20:49:38.298 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.296+0000 7f85eb55d640 1 -- start start 2023-12-16T20:49:38.299 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.296+0000 7f85eb55d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85e40fae30 0x7f85e40feda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:38.299 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.296+0000 7f85eb55d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85e40fc050 0x7f85e40fd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:38.299 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.296+0000 7f85eb55d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e40fd960 con 0x7f85e40fc050 2023-12-16T20:49:38.299 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.296+0000 7f85eb55d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e40fdad0 con 0x7f85e40fae30 2023-12-16T20:49:38.299 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.296+0000 7f85e8af8640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85e40fc050 0x7f85e40fd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:38.299 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.296+0000 7f85e8af8640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85e40fc050 0x7f85e40fd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34496/0 (socket says 172.21.15.52:34496) 2023-12-16T20:49:38.299 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.296+0000 7f85e8af8640 1 -- 172.21.15.52:0/450377648 learned_addr learned my addr 172.21.15.52:0/450377648 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:38.299 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.296+0000 7f85e92f9640 1 --2- 172.21.15.52:0/450377648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85e40fae30 0x7f85e40feda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:38.299 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.296+0000 7f85e8af8640 1 -- 172.21.15.52:0/450377648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85e40fae30 msgr2=0x7f85e40feda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:38.299 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.296+0000 7f85e8af8640 1 --2- 172.21.15.52:0/450377648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85e40fae30 0x7f85e40feda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.299 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.296+0000 7f85e8af8640 1 -- 172.21.15.52:0/450377648 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc005040 con 0x7f85e40fc050 2023-12-16T20:49:38.299 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.297+0000 7f85e8af8640 1 --2- 172.21.15.52:0/450377648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85e40fc050 0x7f85e40fd420 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f85d800c790 tx=0x7f85d80043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:38.299 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.297+0000 7f85e1ffb640 1 -- 172.21.15.52:0/450377648 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85d800a820 con 0x7f85e40fc050 2023-12-16T20:49:38.299 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.297+0000 7f85e1ffb640 1 -- 172.21.15.52:0/450377648 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f85d800ae30 con 0x7f85e40fc050 2023-12-16T20:49:38.300 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.297+0000 7f85e1ffb640 1 -- 172.21.15.52:0/450377648 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85d80113a0 con 0x7f85e40fc050 2023-12-16T20:49:38.300 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.297+0000 7f85eb55d640 1 -- 172.21.15.52:0/450377648 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e40fddb0 con 0x7f85e40fc050 2023-12-16T20:49:38.300 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.297+0000 7f85eb55d640 1 -- 172.21.15.52:0/450377648 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e41148c0 con 0x7f85e40fc050 2023-12-16T20:49:38.301 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.298+0000 7f85eb55d640 1 -- 172.21.15.52:0/450377648 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e41038a0 con 0x7f85e40fc050 2023-12-16T20:49:38.302 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.298+0000 7f85e1ffb640 1 -- 172.21.15.52:0/450377648 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f85d8012400 con 0x7f85e40fc050 2023-12-16T20:49:38.302 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.299+0000 7f85e1ffb640 1 --2- 172.21.15.52:0/450377648 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f85cc068ed0 0x7f85cc06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:38.302 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.299+0000 7f85e1ffb640 1 -- 172.21.15.52:0/450377648 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f85d8086f00 con 0x7f85e40fc050 2023-12-16T20:49:38.302 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.299+0000 7f85e92f9640 1 --2- 172.21.15.52:0/450377648 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f85cc068ed0 0x7f85cc06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:38.302 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.299+0000 7f85e92f9640 1 --2- 172.21.15.52:0/450377648 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f85cc068ed0 0x7f85cc06b380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85dc02bda0 tx=0x7f85dc009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:38.303 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.303+0000 7f85e1ffb640 1 -- 172.21.15.52:0/450377648 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f85d8053420 con 0x7f85e40fc050 2023-12-16T20:49:38.474 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.473+0000 7f85eb55d640 1 -- 172.21.15.52:0/450377648 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f85e4114a50 con 0x7f85e40fc050 2023-12-16T20:49:38.474 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.474+0000 7f85e1ffb640 1 -- 172.21.15.52:0/450377648 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v43) v1 ==== 107+0+11 (secure 0 0 0) 0x7f85d8056a40 con 0x7f85e40fc050 2023-12-16T20:49:38.475 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 32 2023-12-16T20:49:38.476 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.475+0000 7f85eb55d640 1 -- 172.21.15.52:0/450377648 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f85cc068ed0 msgr2=0x7f85cc06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:38.476 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.475+0000 7f85eb55d640 1 --2- 172.21.15.52:0/450377648 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f85cc068ed0 0x7f85cc06b380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85dc02bda0 tx=0x7f85dc009da0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.477 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.476+0000 7f85eb55d640 1 -- 172.21.15.52:0/450377648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85e40fc050 msgr2=0x7f85e40fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:38.477 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.476+0000 7f85eb55d640 1 --2- 172.21.15.52:0/450377648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85e40fc050 0x7f85e40fd420 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f85d800c790 tx=0x7f85d80043c0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.477 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.476+0000 7f85eb55d640 1 -- 172.21.15.52:0/450377648 shutdown_connections 2023-12-16T20:49:38.477 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.476+0000 7f85eb55d640 1 --2- 172.21.15.52:0/450377648 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f85cc068ed0 0x7f85cc06b380 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.477 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.476+0000 7f85eb55d640 1 --2- 172.21.15.52:0/450377648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85e40fc050 0x7f85e40fd420 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.477 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.476+0000 7f85eb55d640 1 --2- 172.21.15.52:0/450377648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85e40fae30 0x7f85e40feda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.477 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.476+0000 7f85eb55d640 1 -- 172.21.15.52:0/450377648 >> 172.21.15.52:0/450377648 conn(0x7f85e40f6580 msgr2=0x7f85e4064130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:38.477 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.476+0000 7f85eb55d640 1 -- 172.21.15.52:0/450377648 shutdown_connections 2023-12-16T20:49:38.477 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.476+0000 7f85eb55d640 1 -- 172.21.15.52:0/450377648 wait complete. 2023-12-16T20:49:38.489 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2023-12-16T20:49:38.596 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.595+0000 7f1a0f15d640 1 -- 172.21.15.52:0/354182776 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a080faed0 msgr2=0x7f1a080fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:38.596 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.595+0000 7f1a0f15d640 1 --2- 172.21.15.52:0/354182776 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a080faed0 0x7f1a080fb2e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009c50 tx=0x7f1a00004d10 comp rx=0 tx=0).stop 2023-12-16T20:49:38.597 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.595+0000 7f1a0f15d640 1 -- 172.21.15.52:0/354182776 shutdown_connections 2023-12-16T20:49:38.597 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.595+0000 7f1a0f15d640 1 --2- 172.21.15.52:0/354182776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a080fc060 0x7f1a080fc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.597 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.595+0000 7f1a0f15d640 1 --2- 172.21.15.52:0/354182776 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a080faed0 0x7f1a080fb2e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.597 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.595+0000 7f1a0f15d640 1 -- 172.21.15.52:0/354182776 >> 172.21.15.52:0/354182776 conn(0x7f1a080f6610 msgr2=0x7f1a080f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:38.597 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.596+0000 7f1a0f15d640 1 -- 172.21.15.52:0/354182776 shutdown_connections 2023-12-16T20:49:38.597 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.596+0000 7f1a0f15d640 1 -- 172.21.15.52:0/354182776 wait complete. 2023-12-16T20:49:38.597 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.596+0000 7f1a0f15d640 1 Processor -- start 2023-12-16T20:49:38.597 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.596+0000 7f1a0f15d640 1 -- start start 2023-12-16T20:49:38.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.596+0000 7f1a0f15d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a080faed0 0x7f1a08065df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:38.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.596+0000 7f1a0f15d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a080fc060 0x7f1a08111530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:38.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.596+0000 7f1a0f15d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a08111a70 con 0x7f1a080faed0 2023-12-16T20:49:38.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.596+0000 7f1a0f15d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a08111bb0 con 0x7f1a080fc060 2023-12-16T20:49:38.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.596+0000 7f1a0cef9640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a080faed0 0x7f1a08065df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:38.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.596+0000 7f1a0cef9640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a080faed0 0x7f1a08065df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34508/0 (socket says 172.21.15.52:34508) 2023-12-16T20:49:38.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.596+0000 7f1a0cef9640 1 -- 172.21.15.52:0/1664221201 learned_addr learned my addr 172.21.15.52:0/1664221201 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:38.600 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.597+0000 7f1a07fff640 1 --2- 172.21.15.52:0/1664221201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a080fc060 0x7f1a08111530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:38.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.597+0000 7f1a0cef9640 1 -- 172.21.15.52:0/1664221201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a080fc060 msgr2=0x7f1a08111530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:38.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.597+0000 7f1a0cef9640 1 --2- 172.21.15.52:0/1664221201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a080fc060 0x7f1a08111530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.597+0000 7f1a0cef9640 1 -- 172.21.15.52:0/1664221201 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a000098b0 con 0x7f1a080faed0 2023-12-16T20:49:38.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.597+0000 7f1a0cef9640 1 --2- 172.21.15.52:0/1664221201 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a080faed0 0x7f1a08065df0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1a00003400 tx=0x7f1a00004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:38.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.597+0000 7f1a057fa640 1 -- 172.21.15.52:0/1664221201 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a0003a070 con 0x7f1a080faed0 2023-12-16T20:49:38.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.597+0000 7f1a057fa640 1 -- 172.21.15.52:0/1664221201 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a000049f0 con 0x7f1a080faed0 2023-12-16T20:49:38.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.597+0000 7f1a0f15d640 1 -- 172.21.15.52:0/1664221201 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a08111e00 con 0x7f1a080faed0 2023-12-16T20:49:38.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.597+0000 7f1a057fa640 1 -- 172.21.15.52:0/1664221201 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a0003e650 con 0x7f1a080faed0 2023-12-16T20:49:38.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.597+0000 7f1a0f15d640 1 -- 172.21.15.52:0/1664221201 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a08112320 con 0x7f1a080faed0 2023-12-16T20:49:38.601 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.598+0000 7f1a0f15d640 1 -- 172.21.15.52:0/1664221201 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a08103860 con 0x7f1a080faed0 2023-12-16T20:49:38.602 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.598+0000 7f1a057fa640 1 -- 172.21.15.52:0/1664221201 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1a00032070 con 0x7f1a080faed0 2023-12-16T20:49:38.602 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.599+0000 7f1a057fa640 1 --2- 172.21.15.52:0/1664221201 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f19f0068ed0 0x7f19f006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:38.602 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.599+0000 7f1a057fa640 1 -- 172.21.15.52:0/1664221201 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1a000a8d30 con 0x7f1a080faed0 2023-12-16T20:49:38.602 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.599+0000 7f1a07fff640 1 --2- 172.21.15.52:0/1664221201 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f19f0068ed0 0x7f19f006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:38.602 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.599+0000 7f1a07fff640 1 --2- 172.21.15.52:0/1664221201 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f19f0068ed0 0x7f19f006b380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f19fc007fd0 tx=0x7f19fc009e90 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:38.604 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.603+0000 7f1a057fa640 1 -- 172.21.15.52:0/1664221201 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1a000750d0 con 0x7f1a080faed0 2023-12-16T20:49:38.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:38.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:38.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:38.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:49:38.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:38 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:38.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:38 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3342902795' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T20:49:38.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:38 smithi052 ceph-mon[80761]: pgmap v131: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.5 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2023-12-16T20:49:38.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:38 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/296626772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-16T20:49:38.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:38 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/450377648' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2023-12-16T20:49:38.774 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.774+0000 7f1a0f15d640 1 -- 172.21.15.52:0/1664221201 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f1a081039f0 con 0x7f1a080faed0 2023-12-16T20:49:38.775 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.774+0000 7f1a057fa640 1 -- 172.21.15.52:0/1664221201 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v43) v1 ==== 107+0+11 (secure 0 0 0) 0x7f1a081039f0 con 0x7f1a080faed0 2023-12-16T20:49:38.775 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 32 2023-12-16T20:49:38.776 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.776+0000 7f1a0f15d640 1 -- 172.21.15.52:0/1664221201 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f19f0068ed0 msgr2=0x7f19f006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:38.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.776+0000 7f1a0f15d640 1 --2- 172.21.15.52:0/1664221201 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f19f0068ed0 0x7f19f006b380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f19fc007fd0 tx=0x7f19fc009e90 comp rx=0 tx=0).stop 2023-12-16T20:49:38.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.776+0000 7f1a0f15d640 1 -- 172.21.15.52:0/1664221201 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a080faed0 msgr2=0x7f1a08065df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:38.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.776+0000 7f1a0f15d640 1 --2- 172.21.15.52:0/1664221201 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a080faed0 0x7f1a08065df0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1a00003400 tx=0x7f1a00004d10 comp rx=0 tx=0).stop 2023-12-16T20:49:38.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.776+0000 7f1a0f15d640 1 -- 172.21.15.52:0/1664221201 shutdown_connections 2023-12-16T20:49:38.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.776+0000 7f1a0f15d640 1 --2- 172.21.15.52:0/1664221201 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f19f0068ed0 0x7f19f006b380 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.776+0000 7f1a0f15d640 1 --2- 172.21.15.52:0/1664221201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a080fc060 0x7f1a08111530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.776+0000 7f1a0f15d640 1 --2- 172.21.15.52:0/1664221201 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a080faed0 0x7f1a08065df0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.776+0000 7f1a0f15d640 1 -- 172.21.15.52:0/1664221201 >> 172.21.15.52:0/1664221201 conn(0x7f1a080f6610 msgr2=0x7f1a080ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:38.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.776+0000 7f1a0f15d640 1 -- 172.21.15.52:0/1664221201 shutdown_connections 2023-12-16T20:49:38.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.776+0000 7f1a0f15d640 1 -- 172.21.15.52:0/1664221201 wait complete. 2023-12-16T20:49:38.787 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T20:49:38.892 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.889+0000 7f5bbe018640 1 -- 172.21.15.52:0/3444486494 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bb80ff3b0 msgr2=0x7f5bb8101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:38.892 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.889+0000 7f5bbe018640 1 --2- 172.21.15.52:0/3444486494 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bb80ff3b0 0x7f5bb8101790 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5ba00048d0 tx=0x7f5ba0009da0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.892 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.889+0000 7f5bbe018640 1 -- 172.21.15.52:0/3444486494 shutdown_connections 2023-12-16T20:49:38.892 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.889+0000 7f5bbe018640 1 --2- 172.21.15.52:0/3444486494 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bb80ff3b0 0x7f5bb8101790 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.892 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.889+0000 7f5bbe018640 1 --2- 172.21.15.52:0/3444486494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bb80fca90 0x7f5bb80fee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.892 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.889+0000 7f5bbe018640 1 -- 172.21.15.52:0/3444486494 >> 172.21.15.52:0/3444486494 conn(0x7f5bb80f6510 msgr2=0x7f5bb80f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:38.893 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.889+0000 7f5bbe018640 1 -- 172.21.15.52:0/3444486494 shutdown_connections 2023-12-16T20:49:38.893 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.889+0000 7f5bbe018640 1 -- 172.21.15.52:0/3444486494 wait complete. 2023-12-16T20:49:38.893 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.890+0000 7f5bbe018640 1 Processor -- start 2023-12-16T20:49:38.893 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.890+0000 7f5bbe018640 1 -- start start 2023-12-16T20:49:38.893 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.890+0000 7f5bbe018640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bb80fca90 0x7f5bb810c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:38.893 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.890+0000 7f5bbe018640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bb80ff3b0 0x7f5bb810cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:38.893 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.890+0000 7f5bbe018640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb810d3c0 con 0x7f5bb80fca90 2023-12-16T20:49:38.893 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.890+0000 7f5bbe018640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb8111d80 con 0x7f5bb80ff3b0 2023-12-16T20:49:38.894 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.890+0000 7f5bb77fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bb80fca90 0x7f5bb810c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:38.894 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.890+0000 7f5bb77fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bb80fca90 0x7f5bb810c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34516/0 (socket says 172.21.15.52:34516) 2023-12-16T20:49:38.894 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.890+0000 7f5bb77fe640 1 -- 172.21.15.52:0/2207713090 learned_addr learned my addr 172.21.15.52:0/2207713090 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:38.894 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.891+0000 7f5bb6ffd640 1 --2- 172.21.15.52:0/2207713090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bb80ff3b0 0x7f5bb810cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:38.894 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.891+0000 7f5bb77fe640 1 -- 172.21.15.52:0/2207713090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bb80ff3b0 msgr2=0x7f5bb810cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:38.894 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.891+0000 7f5bb77fe640 1 --2- 172.21.15.52:0/2207713090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bb80ff3b0 0x7f5bb810cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:38.894 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.891+0000 7f5bb77fe640 1 -- 172.21.15.52:0/2207713090 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba0005040 con 0x7f5bb80fca90 2023-12-16T20:49:38.894 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.891+0000 7f5bb6ffd640 1 --2- 172.21.15.52:0/2207713090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bb80ff3b0 0x7f5bb810cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T20:49:38.894 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.891+0000 7f5bb77fe640 1 --2- 172.21.15.52:0/2207713090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bb80fca90 0x7f5bb810c7a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5ba800bbb0 tx=0x7f5ba8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:38.894 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.891+0000 7f5baffff640 1 -- 172.21.15.52:0/2207713090 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ba800c9c0 con 0x7f5bb80fca90 2023-12-16T20:49:38.894 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.891+0000 7f5baffff640 1 -- 172.21.15.52:0/2207713090 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ba80092e0 con 0x7f5bb80fca90 2023-12-16T20:49:38.895 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.891+0000 7f5bbe018640 1 -- 172.21.15.52:0/2207713090 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb8111f80 con 0x7f5bb80fca90 2023-12-16T20:49:38.896 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.891+0000 7f5baffff640 1 -- 172.21.15.52:0/2207713090 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ba8007640 con 0x7f5bb80fca90 2023-12-16T20:49:38.897 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.891+0000 7f5bbe018640 1 -- 172.21.15.52:0/2207713090 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb81123c0 con 0x7f5bb80fca90 2023-12-16T20:49:38.898 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.892+0000 7f5bbe018640 1 -- 172.21.15.52:0/2207713090 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb8106960 con 0x7f5bb80fca90 2023-12-16T20:49:38.898 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.892+0000 7f5baffff640 1 -- 172.21.15.52:0/2207713090 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f5ba8012070 con 0x7f5bb80fca90 2023-12-16T20:49:38.898 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.893+0000 7f5baffff640 1 --2- 172.21.15.52:0/2207713090 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5ba4068ed0 0x7f5ba406b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:38.898 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.893+0000 7f5bb6ffd640 1 --2- 172.21.15.52:0/2207713090 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5ba4068ed0 0x7f5ba406b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:38.899 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.893+0000 7f5baffff640 1 -- 172.21.15.52:0/2207713090 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5ba8086e20 con 0x7f5bb80fca90 2023-12-16T20:49:38.899 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.893+0000 7f5bb6ffd640 1 --2- 172.21.15.52:0/2207713090 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5ba4068ed0 0x7f5ba406b380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5ba00315d0 tx=0x7f5ba0009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:38.899 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:38.897+0000 7f5baffff640 1 -- 172.21.15.52:0/2207713090 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5ba8053220 con 0x7f5bb80fca90 2023-12-16T20:49:38.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:38.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:38.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:49:38.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:49:38.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:38 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:38.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:38 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/3342902795' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T20:49:38.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:38 smithi150 ceph-mon[88000]: pgmap v131: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.5 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2023-12-16T20:49:38.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:38 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/296626772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-16T20:49:38.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:38 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/450377648' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2023-12-16T20:49:39.106 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:39.105+0000 7f5bbe018640 1 -- 172.21.15.52:0/2207713090 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bb81049b0 con 0x7f5bb80fca90 2023-12-16T20:49:39.106 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:39.106+0000 7f5baffff640 1 -- 172.21.15.52:0/2207713090 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4722 (secure 0 0 0) 0x7f5ba8056840 con 0x7f5bb80fca90 2023-12-16T20:49:39.107 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-12-16T20:49:39.107 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-12-16T20:49:20.450316+0000","modified":"2023-12-16T20:49:34.145328+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14486,"mds_1":24263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14486":{"gid":14486,"name":"cephfs.smithi052.szlsle","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.52:6829/1781554610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":1781554610},{"type":"v1","addr":"172.21.15.52:6829","nonce":1781554610}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24263":{"gid":24263,"name":"cephfs.smithi150.knuivd","rank":1,"incarnation":8,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6825/351844450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":351844450},{"type":"v1","addr":"172.21.15.150:6825","nonce":351844450}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24279":{"gid":24279,"name":"cephfs.smithi150.njrgli","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.150:6827/4088837116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4088837116},{"type":"v1","addr":"172.21.15.150:6827","nonce":4088837116}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24287":{"gid":24287,"name":"cephfs.smithi052.bvmosa","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6827/2170358701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":2170358701},{"type":"v1","addr":"172.21.15.52:6827","nonce":2170358701}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2023-12-16T20:49:39.108 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 19 2023-12-16T20:49:39.109 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:39.107+0000 7f5bbe018640 1 -- 172.21.15.52:0/2207713090 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5ba4068ed0 msgr2=0x7f5ba406b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:39.109 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:39.107+0000 7f5bbe018640 1 --2- 172.21.15.52:0/2207713090 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5ba4068ed0 0x7f5ba406b380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5ba00315d0 tx=0x7f5ba0009da0 comp rx=0 tx=0).stop 2023-12-16T20:49:39.109 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:39.107+0000 7f5bbe018640 1 -- 172.21.15.52:0/2207713090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bb80fca90 msgr2=0x7f5bb810c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:39.109 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:39.107+0000 7f5bbe018640 1 --2- 172.21.15.52:0/2207713090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bb80fca90 0x7f5bb810c7a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5ba800bbb0 tx=0x7f5ba8009840 comp rx=0 tx=0).stop 2023-12-16T20:49:39.109 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:39.107+0000 7f5bbe018640 1 -- 172.21.15.52:0/2207713090 shutdown_connections 2023-12-16T20:49:39.109 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:39.107+0000 7f5bbe018640 1 --2- 172.21.15.52:0/2207713090 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5ba4068ed0 0x7f5ba406b380 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:39.109 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:39.107+0000 7f5bbe018640 1 --2- 172.21.15.52:0/2207713090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bb80ff3b0 0x7f5bb810cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:39.109 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:39.107+0000 7f5bbe018640 1 --2- 172.21.15.52:0/2207713090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bb80fca90 0x7f5bb810c7a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:39.109 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:39.107+0000 7f5bbe018640 1 -- 172.21.15.52:0/2207713090 >> 172.21.15.52:0/2207713090 conn(0x7f5bb80f6510 msgr2=0x7f5bb80f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:39.109 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:39.108+0000 7f5bbe018640 1 -- 172.21.15.52:0/2207713090 shutdown_connections 2023-12-16T20:49:39.109 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:39.108+0000 7f5bbe018640 1 -- 172.21.15.52:0/2207713090 wait complete. 2023-12-16T20:49:39.119 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 19, 'max_mds': 2, 'flags': 50} 2023-12-16T20:49:39.119 INFO:teuthology.run_tasks:Running task kclient... 2023-12-16T20:49:39.179 INFO:tasks.kclient:Mounting kernel clients... 2023-12-16T20:49:39.180 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2023-12-16T20:49:39.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:39.180 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list 2023-12-16T20:49:39.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:39.196 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link delete ceph-brx 2023-12-16T20:49:39.269 INFO:teuthology.orchestra.run.smithi052.stderr:Cannot find device "ceph-brx" 2023-12-16T20:49:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T20:49:39.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:39.271 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2023-12-16T20:49:39.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:39.290 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link delete ceph-brx 2023-12-16T20:49:39.366 INFO:teuthology.orchestra.run.smithi150.stderr:Cannot find device "ceph-brx" 2023-12-16T20:49:39.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T20:49:39.368 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-12-16T20:49:39.368 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-16T20:49:39.368 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-16T20:49:39.369 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph fs ls 2023-12-16T20:49:39.637 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:39.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:39 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/1664221201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2023-12-16T20:49:39.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:39 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/2207713090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-16T20:49:39.726 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:39 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/1664221201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2023-12-16T20:49:39.726 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:39 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/2207713090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-16T20:49:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.569+0000 7f66171ca700 1 -- 172.21.15.52:0/469206147 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6610103590 msgr2=0x7f66100fcc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.569+0000 7f66171ca700 1 --2- 172.21.15.52:0/469206147 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6610103590 0x7f66100fcc00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f65fc004940 tx=0x7f65fc009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.569+0000 7f66171ca700 1 -- 172.21.15.52:0/469206147 shutdown_connections 2023-12-16T20:49:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.569+0000 7f66171ca700 1 --2- 172.21.15.52:0/469206147 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6610103590 0x7f66100fcc00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.569+0000 7f66171ca700 1 --2- 172.21.15.52:0/469206147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6610102bf0 0x7f6610102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.569+0000 7f66171ca700 1 -- 172.21.15.52:0/469206147 >> 172.21.15.52:0/469206147 conn(0x7f66100f87a0 msgr2=0x7f66100fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.569+0000 7f66171ca700 1 -- 172.21.15.52:0/469206147 shutdown_connections 2023-12-16T20:49:40.571 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.569+0000 7f66171ca700 1 -- 172.21.15.52:0/469206147 wait complete. 2023-12-16T20:49:40.572 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.570+0000 7f66171ca700 1 Processor -- start 2023-12-16T20:49:40.572 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.570+0000 7f66171ca700 1 -- start start 2023-12-16T20:49:40.572 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.570+0000 7f66171ca700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6610102bf0 0x7f6610113660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:40.572 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.570+0000 7f66171ca700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6610103590 0x7f6610113be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:40.572 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.570+0000 7f66171ca700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661010d520 con 0x7f6610103590 2023-12-16T20:49:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.570+0000 7f66171ca700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661010d690 con 0x7f6610102bf0 2023-12-16T20:49:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.571+0000 7f660ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6610103590 0x7f6610113be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.571+0000 7f660ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6610103590 0x7f6610113be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34546/0 (socket says 172.21.15.52:34546) 2023-12-16T20:49:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.571+0000 7f660ffff700 1 -- 172.21.15.52:0/695294865 learned_addr learned my addr 172.21.15.52:0/695294865 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.571+0000 7f660ffff700 1 -- 172.21.15.52:0/695294865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6610102bf0 msgr2=0x7f6610113660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.571+0000 7f6614f66700 1 --2- 172.21.15.52:0/695294865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6610102bf0 0x7f6610113660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.571+0000 7f660ffff700 1 --2- 172.21.15.52:0/695294865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6610102bf0 0x7f6610113660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.571+0000 7f660ffff700 1 -- 172.21.15.52:0/695294865 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65fc005040 con 0x7f6610103590 2023-12-16T20:49:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.571+0000 7f660ffff700 1 --2- 172.21.15.52:0/695294865 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6610103590 0x7f6610113be0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009180 tx=0x7f65fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.571+0000 7f660d7fa700 1 -- 172.21.15.52:0/695294865 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65fc00e040 con 0x7f6610103590 2023-12-16T20:49:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.571+0000 7f66171ca700 1 -- 172.21.15.52:0/695294865 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661010d910 con 0x7f6610103590 2023-12-16T20:49:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.571+0000 7f660d7fa700 1 -- 172.21.15.52:0/695294865 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65fc004290 con 0x7f6610103590 2023-12-16T20:49:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.571+0000 7f660d7fa700 1 -- 172.21.15.52:0/695294865 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65fc01c3f0 con 0x7f6610103590 2023-12-16T20:49:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.571+0000 7f66171ca700 1 -- 172.21.15.52:0/695294865 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661010de50 con 0x7f6610103590 2023-12-16T20:49:40.574 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.572+0000 7f66171ca700 1 -- 172.21.15.52:0/695294865 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66100fff00 con 0x7f6610103590 2023-12-16T20:49:40.574 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.573+0000 7f660d7fa700 1 -- 172.21.15.52:0/695294865 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f65fc00bc50 con 0x7f6610103590 2023-12-16T20:49:40.574 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.573+0000 7f660d7fa700 1 --2- 172.21.15.52:0/695294865 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6600068f70 0x7f660006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:40.575 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.573+0000 7f660d7fa700 1 -- 172.21.15.52:0/695294865 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f65fc088030 con 0x7f6610103590 2023-12-16T20:49:40.575 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.573+0000 7f6614f66700 1 --2- 172.21.15.52:0/695294865 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6600068f70 0x7f660006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:40.575 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.574+0000 7f6614f66700 1 --2- 172.21.15.52:0/695294865 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6600068f70 0x7f660006b420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6604006fd0 tx=0x7f6604008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:40.578 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.578+0000 7f660d7fa700 1 -- 172.21.15.52:0/695294865 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f65fc058090 con 0x7f6610103590 2023-12-16T20:49:40.732 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:40 smithi052 ceph-mon[80761]: pgmap v132: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.2 KiB/s rd, 0 B/s wr, 8 op/s 2023-12-16T20:49:40.822 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.821+0000 7f66171ca700 1 -- 172.21.15.52:0/695294865 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f661004bb80 con 0x7f6610103590 2023-12-16T20:49:40.822 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.822+0000 7f660d7fa700 1 -- 172.21.15.52:0/695294865 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+83 (secure 0 0 0) 0x7f65fc057c20 con 0x7f6610103590 2023-12-16T20:49:40.822 INFO:teuthology.orchestra.run.smithi052.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2023-12-16T20:49:40.824 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.824+0000 7f66171ca700 1 -- 172.21.15.52:0/695294865 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6600068f70 msgr2=0x7f660006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:40.825 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.824+0000 7f66171ca700 1 --2- 172.21.15.52:0/695294865 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6600068f70 0x7f660006b420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6604006fd0 tx=0x7f6604008040 comp rx=0 tx=0).stop 2023-12-16T20:49:40.825 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.824+0000 7f66171ca700 1 -- 172.21.15.52:0/695294865 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6610103590 msgr2=0x7f6610113be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:40.825 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.824+0000 7f66171ca700 1 --2- 172.21.15.52:0/695294865 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6610103590 0x7f6610113be0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009180 tx=0x7f65fc009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:40.825 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.824+0000 7f66171ca700 1 -- 172.21.15.52:0/695294865 shutdown_connections 2023-12-16T20:49:40.825 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.824+0000 7f66171ca700 1 --2- 172.21.15.52:0/695294865 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f6600068f70 0x7f660006b420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:40.825 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.824+0000 7f66171ca700 1 --2- 172.21.15.52:0/695294865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6610102bf0 0x7f6610113660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:40.825 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.824+0000 7f66171ca700 1 --2- 172.21.15.52:0/695294865 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6610103590 0x7f6610113be0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:40.825 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.824+0000 7f66171ca700 1 -- 172.21.15.52:0/695294865 >> 172.21.15.52:0/695294865 conn(0x7f66100f87a0 msgr2=0x7f66100fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:40.825 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.824+0000 7f66171ca700 1 -- 172.21.15.52:0/695294865 shutdown_connections 2023-12-16T20:49:40.825 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:40.824+0000 7f66171ca700 1 -- 172.21.15.52:0/695294865 wait complete. 2023-12-16T20:49:40.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:40 smithi150 ceph-mon[88000]: pgmap v132: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.2 KiB/s rd, 0 B/s wr, 8 op/s 2023-12-16T20:49:41.177 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T20:49:41.177 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T20:49:41.177 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi052.front.sepia.ceph.com 2023-12-16T20:49:41.177 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-16T20:49:41.177 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-16T20:49:41.177 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T20:49:41.177 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T20:49:41.177 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T20:49:41.178 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T20:49:41.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:41.178 DEBUG:teuthology.orchestra.run.smithi052:> ip addr 2023-12-16T20:49:41.194 INFO:teuthology.orchestra.run.smithi052.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T20:49:41.194 INFO:teuthology.orchestra.run.smithi052.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T20:49:41.194 INFO:teuthology.orchestra.run.smithi052.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T20:49:41.194 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2023-12-16T20:49:41.194 INFO:teuthology.orchestra.run.smithi052.stdout: inet6 ::1/128 scope host 2023-12-16T20:49:41.194 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2023-12-16T20:49:41.194 INFO:teuthology.orchestra.run.smithi052.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T20:49:41.194 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:6b:fd:d2 brd ff:ff:ff:ff:ff:ff 2023-12-16T20:49:41.194 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp7s0f0 2023-12-16T20:49:41.194 INFO:teuthology.orchestra.run.smithi052.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T20:49:41.195 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:6b:fd:d3 brd ff:ff:ff:ff:ff:ff 2023-12-16T20:49:41.195 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp7s0f1 2023-12-16T20:49:41.195 INFO:teuthology.orchestra.run.smithi052.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T20:49:41.195 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:bd:15:de brd ff:ff:ff:ff:ff:ff 2023-12-16T20:49:41.195 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp1s0f0 2023-12-16T20:49:41.195 INFO:teuthology.orchestra.run.smithi052.stdout: inet 172.21.15.52/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-12-16T20:49:41.195 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft 42358sec preferred_lft 42358sec 2023-12-16T20:49:41.195 INFO:teuthology.orchestra.run.smithi052.stdout: inet6 fe80::ec4:7aff:febd:15de/64 scope link noprefixroute 2023-12-16T20:49:41.195 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2023-12-16T20:49:41.195 INFO:teuthology.orchestra.run.smithi052.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T20:49:41.195 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:bd:15:df brd ff:ff:ff:ff:ff:ff 2023-12-16T20:49:41.195 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp1s0f1 2023-12-16T20:49:41.196 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-16T20:49:41.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:41.196 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec bash -c ' 2023-12-16T20:49:41.196 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2023-12-16T20:49:41.196 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link add name ceph-brx type bridge 2023-12-16T20:49:41.196 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip addr flush dev ceph-brx 2023-12-16T20:49:41.196 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link set ceph-brx up 2023-12-16T20:49:41.196 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-16T20:49:41.196 DEBUG:teuthology.orchestra.run.smithi052:> ') 2023-12-16T20:49:41.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:41.380 DEBUG:teuthology.orchestra.run.smithi052:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T20:49:41.411 INFO:teuthology.orchestra.run.smithi052.stdout:1 2023-12-16T20:49:41.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:41.413 DEBUG:teuthology.orchestra.run.smithi052:> route 2023-12-16T20:49:41.485 INFO:teuthology.orchestra.run.smithi052.stdout:Kernel IP routing table 2023-12-16T20:49:41.485 INFO:teuthology.orchestra.run.smithi052.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T20:49:41.485 INFO:teuthology.orchestra.run.smithi052.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-12-16T20:49:41.486 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-12-16T20:49:41.486 INFO:teuthology.orchestra.run.smithi052.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T20:49:41.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:41.486 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec bash -c ' 2023-12-16T20:49:41.486 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2023-12-16T20:49:41.486 DEBUG:teuthology.orchestra.run.smithi052:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-16T20:49:41.486 DEBUG:teuthology.orchestra.run.smithi052:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-16T20:49:41.486 DEBUG:teuthology.orchestra.run.smithi052:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-16T20:49:41.487 DEBUG:teuthology.orchestra.run.smithi052:> ') 2023-12-16T20:49:41.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:41.765 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list 2023-12-16T20:49:41.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:41.784 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list-id 2023-12-16T20:49:41.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:41.840 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec bash -c ' 2023-12-16T20:49:41.840 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2023-12-16T20:49:41.840 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T20:49:41.840 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-12-16T20:49:41.840 DEBUG:teuthology.orchestra.run.smithi052:> ') 2023-12-16T20:49:41.909 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:41 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/695294865' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-16T20:49:41.930 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-12-16T20:49:41.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:41.931 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec bash -c ' 2023-12-16T20:49:41.931 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2023-12-16T20:49:41.931 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-12-16T20:49:41.931 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-16T20:49:41.931 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-12-16T20:49:41.931 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-12-16T20:49:41.931 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-12-16T20:49:41.931 DEBUG:teuthology.orchestra.run.smithi052:> ') 2023-12-16T20:49:41.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:41 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/695294865' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-16T20:49:42.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:42.088 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec bash -c ' 2023-12-16T20:49:42.088 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2023-12-16T20:49:42.088 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link set brx.0 up 2023-12-16T20:49:42.088 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link set dev brx.0 master ceph-brx 2023-12-16T20:49:42.088 DEBUG:teuthology.orchestra.run.smithi052:> ') 2023-12-16T20:49:42.137 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T20:49:42.137 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-16T20:49:42.152 INFO:teuthology.orchestra.run.smithi052.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-16T20:49:42.152 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T20:49:42.152 DEBUG:teuthology.orchestra.run.smithi052:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-16T20:49:42.205 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-12-16T20:49:42.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:42.206 DEBUG:teuthology.orchestra.run.smithi052:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-16T20:49:42.293 INFO:teuthology.orchestra.run.smithi052.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-16T20:49:42.294 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-12-16T20:49:42.294 INFO:teuthology.orchestra.run.smithi052.stdout:invalid new device string format 2023-12-16T20:49:42.399 INFO:teuthology.orchestra.run.smithi052.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-16T20:49:42.399 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-12-16T20:49:42.399 INFO:teuthology.orchestra.run.smithi052.stdout:invalid new device string format 2023-12-16T20:49:42.399 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: resolved to: "172.21.15.52:3300,172.21.15.150:3300" 2023-12-16T20:49:42.399 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-16T20:49:42.404 INFO:teuthology.orchestra.run.smithi052.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-16T20:49:42.404 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-12-16T20:49:42.404 INFO:teuthology.orchestra.run.smithi052.stdout:invalid new device string format 2023-12-16T20:49:42.404 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: resolved to: "172.21.15.52:3300,172.21.15.150:3300" 2023-12-16T20:49:42.404 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-16T20:49:42.404 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: resolved to: "172.21.15.52:6789,172.21.15.150:6789" 2023-12-16T20:49:42.404 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.52:6789,172.21.15.150:6789:/ 2023-12-16T20:49:42.404 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=cf043a8a-9c53-11ee-95a5-87774f69a715" will pass to kernel 2023-12-16T20:49:42.405 INFO:teuthology.orchestra.run.smithi052.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-12-16T20:49:42.405 INFO:teuthology.orchestra.run.smithi052.stdout: You just mounted a file system that supports labels which does not 2023-12-16T20:49:42.405 INFO:teuthology.orchestra.run.smithi052.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-12-16T20:49:42.405 INFO:teuthology.orchestra.run.smithi052.stdout: applications will generate AVC messages and not be allowed access to 2023-12-16T20:49:42.405 INFO:teuthology.orchestra.run.smithi052.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-12-16T20:49:42.407 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-12-16T20:49:42.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:42.407 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-16T20:49:42.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:42.439 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-16T20:49:42.500 INFO:teuthology.orchestra.run.smithi052.stdout:client24307 2023-12-16T20:49:42.501 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:49:42.501 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T20:49:42.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:42.558 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-16T20:49:42.620 INFO:teuthology.orchestra.run.smithi052.stdout:client24307 2023-12-16T20:49:42.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:42.620 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-16T20:49:42.681 INFO:teuthology.orchestra.run.smithi052.stdout:client24307 2023-12-16T20:49:42.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:42.681 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715.client24307/status') 2023-12-16T20:49:42.754 INFO:teuthology.orchestra.run.smithi052.stdout:instance: client.24307 (3)172.21.15.52:0/2150243870 2023-12-16T20:49:42.755 INFO:teuthology.orchestra.run.smithi052.stdout:blocklisted: false 2023-12-16T20:49:42.755 INFO:teuthology.orchestra.run.smithi052.stderr:0+1 records in 2023-12-16T20:49:42.755 INFO:teuthology.orchestra.run.smithi052.stderr:0+1 records out 2023-12-16T20:49:42.755 INFO:teuthology.orchestra.run.smithi052.stderr:71 bytes copied, 2.4202e-05 s, 2.9 MB/s 2023-12-16T20:49:42.756 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2023-12-16T20:49:42.756 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-16T20:49:42.756 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-16T20:49:42.756 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -- ceph fs ls 2023-12-16T20:49:42.824 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:42 smithi052 ceph-mon[80761]: pgmap v133: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 7 op/s 2023-12-16T20:49:42.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:42 smithi150 ceph-mon[88000]: pgmap v133: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 7 op/s 2023-12-16T20:49:43.065 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:43.988 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.986+0000 7fdb965a8700 1 -- 172.21.15.52:0/251154923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb90101b00 msgr2=0x7fdb90101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:43.988 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.986+0000 7fdb965a8700 1 --2- 172.21.15.52:0/251154923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb90101b00 0x7fdb90101ed0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdb80004940 tx=0x7fdb80009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:43.988 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.986+0000 7fdb965a8700 1 -- 172.21.15.52:0/251154923 shutdown_connections 2023-12-16T20:49:43.988 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.986+0000 7fdb965a8700 1 --2- 172.21.15.52:0/251154923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb900fbb40 0x7fdb900fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:43.989 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.986+0000 7fdb965a8700 1 --2- 172.21.15.52:0/251154923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb90101b00 0x7fdb90101ed0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:43.989 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.986+0000 7fdb965a8700 1 -- 172.21.15.52:0/251154923 >> 172.21.15.52:0/251154923 conn(0x7fdb900f76c0 msgr2=0x7fdb900f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:43.989 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.986+0000 7fdb965a8700 1 -- 172.21.15.52:0/251154923 shutdown_connections 2023-12-16T20:49:43.989 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.986+0000 7fdb965a8700 1 -- 172.21.15.52:0/251154923 wait complete. 2023-12-16T20:49:43.989 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.986+0000 7fdb965a8700 1 Processor -- start 2023-12-16T20:49:43.989 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.987+0000 7fdb965a8700 1 -- start start 2023-12-16T20:49:43.989 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.987+0000 7fdb965a8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb900fbb40 0x7fdb90113670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:43.989 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.987+0000 7fdb965a8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb90101b00 0x7fdb90113bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:43.989 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.987+0000 7fdb965a8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb90114240 con 0x7fdb900fbb40 2023-12-16T20:49:43.989 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.987+0000 7fdb965a8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9010d530 con 0x7fdb90101b00 2023-12-16T20:49:43.989 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.987+0000 7fdb8ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb900fbb40 0x7fdb90113670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:43.989 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.987+0000 7fdb8ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb900fbb40 0x7fdb90113670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41834/0 (socket says 172.21.15.52:41834) 2023-12-16T20:49:43.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.987+0000 7fdb8ffff700 1 -- 172.21.15.52:0/4253417414 learned_addr learned my addr 172.21.15.52:0/4253417414 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:43.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.987+0000 7fdb8f7fe700 1 --2- 172.21.15.52:0/4253417414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb90101b00 0x7fdb90113bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:43.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.987+0000 7fdb8ffff700 1 -- 172.21.15.52:0/4253417414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb90101b00 msgr2=0x7fdb90113bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:43.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.987+0000 7fdb8ffff700 1 --2- 172.21.15.52:0/4253417414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb90101b00 0x7fdb90113bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:43.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.987+0000 7fdb8ffff700 1 -- 172.21.15.52:0/4253417414 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb80005040 con 0x7fdb900fbb40 2023-12-16T20:49:43.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.987+0000 7fdb8f7fe700 1 --2- 172.21.15.52:0/4253417414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb90101b00 0x7fdb90113bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T20:49:43.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.987+0000 7fdb8ffff700 1 --2- 172.21.15.52:0/4253417414 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb900fbb40 0x7fdb90113670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdb80009360 tx=0x7fdb80009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:43.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.988+0000 7fdb8cff9700 1 -- 172.21.15.52:0/4253417414 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb8000e040 con 0x7fdb900fbb40 2023-12-16T20:49:43.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.988+0000 7fdb8cff9700 1 -- 172.21.15.52:0/4253417414 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb8000bc50 con 0x7fdb900fbb40 2023-12-16T20:49:43.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.988+0000 7fdb8cff9700 1 -- 172.21.15.52:0/4253417414 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb8001c630 con 0x7fdb900fbb40 2023-12-16T20:49:43.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.988+0000 7fdb965a8700 1 -- 172.21.15.52:0/4253417414 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9010d760 con 0x7fdb900fbb40 2023-12-16T20:49:43.990 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.988+0000 7fdb965a8700 1 -- 172.21.15.52:0/4253417414 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb9010dca0 con 0x7fdb900fbb40 2023-12-16T20:49:43.991 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.989+0000 7fdb965a8700 1 -- 172.21.15.52:0/4253417414 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb900fb2c0 con 0x7fdb900fbb40 2023-12-16T20:49:43.991 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.989+0000 7fdb8cff9700 1 -- 172.21.15.52:0/4253417414 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fdb80013420 con 0x7fdb900fbb40 2023-12-16T20:49:43.991 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.990+0000 7fdb8cff9700 1 --2- 172.21.15.52:0/4253417414 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fdb7c068f70 0x7fdb7c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:43.991 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.990+0000 7fdb8cff9700 1 -- 172.21.15.52:0/4253417414 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fdb800880f0 con 0x7fdb900fbb40 2023-12-16T20:49:43.992 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.990+0000 7fdb8f7fe700 1 --2- 172.21.15.52:0/4253417414 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fdb7c068f70 0x7fdb7c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:43.992 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.990+0000 7fdb8f7fe700 1 --2- 172.21.15.52:0/4253417414 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fdb7c068f70 0x7fdb7c06b420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdb78007fd0 tx=0x7fdb78009040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:43.995 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:43.994+0000 7fdb8cff9700 1 -- 172.21.15.52:0/4253417414 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdb80058000 con 0x7fdb900fbb40 2023-12-16T20:49:44.236 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:44.236+0000 7fdb965a8700 1 -- 172.21.15.52:0/4253417414 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdb9010ecf0 con 0x7fdb900fbb40 2023-12-16T20:49:44.237 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:44.236+0000 7fdb8cff9700 1 -- 172.21.15.52:0/4253417414 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+83 (secure 0 0 0) 0x7fdb80057b90 con 0x7fdb900fbb40 2023-12-16T20:49:44.237 INFO:teuthology.orchestra.run.smithi052.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2023-12-16T20:49:44.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:44.238+0000 7fdb965a8700 1 -- 172.21.15.52:0/4253417414 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fdb7c068f70 msgr2=0x7fdb7c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:44.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:44.238+0000 7fdb965a8700 1 --2- 172.21.15.52:0/4253417414 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fdb7c068f70 0x7fdb7c06b420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdb78007fd0 tx=0x7fdb78009040 comp rx=0 tx=0).stop 2023-12-16T20:49:44.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:44.238+0000 7fdb965a8700 1 -- 172.21.15.52:0/4253417414 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb900fbb40 msgr2=0x7fdb90113670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:44.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:44.238+0000 7fdb965a8700 1 --2- 172.21.15.52:0/4253417414 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb900fbb40 0x7fdb90113670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdb80009360 tx=0x7fdb80009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:44.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:44.238+0000 7fdb965a8700 1 -- 172.21.15.52:0/4253417414 shutdown_connections 2023-12-16T20:49:44.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:44.238+0000 7fdb965a8700 1 --2- 172.21.15.52:0/4253417414 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fdb7c068f70 0x7fdb7c06b420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:44.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:44.238+0000 7fdb965a8700 1 --2- 172.21.15.52:0/4253417414 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb900fbb40 0x7fdb90113670 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:44.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:44.238+0000 7fdb965a8700 1 --2- 172.21.15.52:0/4253417414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb90101b00 0x7fdb90113bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:44.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:44.238+0000 7fdb965a8700 1 -- 172.21.15.52:0/4253417414 >> 172.21.15.52:0/4253417414 conn(0x7fdb900f76c0 msgr2=0x7fdb90065d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:44.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:44.238+0000 7fdb965a8700 1 -- 172.21.15.52:0/4253417414 shutdown_connections 2023-12-16T20:49:44.240 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:44.238+0000 7fdb965a8700 1 -- 172.21.15.52:0/4253417414 wait complete. 2023-12-16T20:49:44.559 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T20:49:44.559 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T20:49:44.559 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2023-12-16T20:49:44.559 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-12-16T20:49:44.559 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-16T20:49:44.559 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T20:49:44.560 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T20:49:44.560 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T20:49:44.560 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-12-16T20:49:44.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:44.560 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2023-12-16T20:49:44.576 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T20:49:44.577 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T20:49:44.577 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T20:49:44.577 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2023-12-16T20:49:44.577 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2023-12-16T20:49:44.577 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2023-12-16T20:49:44.577 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T20:49:44.577 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2023-12-16T20:49:44.577 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2023-12-16T20:49:44.577 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T20:49:44.577 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2023-12-16T20:49:44.577 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2023-12-16T20:49:44.577 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T20:49:44.578 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2023-12-16T20:49:44.578 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T20:49:44.578 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2023-12-16T20:49:44.578 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-16T20:49:44.578 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42360sec preferred_lft 42360sec 2023-12-16T20:49:44.578 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::ec4:7aff:fe88:6e35/64 scope link 2023-12-16T20:49:44.578 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2023-12-16T20:49:44.578 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-16T20:49:44.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:44.578 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec bash -c ' 2023-12-16T20:49:44.579 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2023-12-16T20:49:44.579 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add name ceph-brx type bridge 2023-12-16T20:49:44.579 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr flush dev ceph-brx 2023-12-16T20:49:44.579 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set ceph-brx up 2023-12-16T20:49:44.579 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-16T20:49:44.579 DEBUG:teuthology.orchestra.run.smithi150:> ') 2023-12-16T20:49:44.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:44.740 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T20:49:44.766 INFO:teuthology.orchestra.run.smithi150.stdout:1 2023-12-16T20:49:44.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:44.768 DEBUG:teuthology.orchestra.run.smithi150:> route 2023-12-16T20:49:44.842 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2023-12-16T20:49:44.842 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T20:49:44.842 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-16T20:49:44.842 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-16T20:49:44.842 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T20:49:44.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:44.842 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec bash -c ' 2023-12-16T20:49:44.842 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2023-12-16T20:49:44.842 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-16T20:49:44.843 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-16T20:49:44.843 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-16T20:49:44.843 DEBUG:teuthology.orchestra.run.smithi150:> ') 2023-12-16T20:49:45.011 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:44 smithi150 ceph-mon[88000]: pgmap v134: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.4 KiB/s rd, 170 B/s wr, 8 op/s 2023-12-16T20:49:45.012 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:44 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/4253417414' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-16T20:49:45.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:45.106 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2023-12-16T20:49:45.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:45.121 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list-id 2023-12-16T20:49:45.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:45.176 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec bash -c ' 2023-12-16T20:49:45.176 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2023-12-16T20:49:45.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T20:49:45.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-12-16T20:49:45.176 DEBUG:teuthology.orchestra.run.smithi150:> ') 2023-12-16T20:49:45.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:44 smithi052 ceph-mon[80761]: pgmap v134: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.4 KiB/s rd, 170 B/s wr, 8 op/s 2023-12-16T20:49:45.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:44 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/4253417414' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-16T20:49:45.262 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-12-16T20:49:45.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:45.263 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec bash -c ' 2023-12-16T20:49:45.263 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2023-12-16T20:49:45.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-12-16T20:49:45.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-16T20:49:45.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-12-16T20:49:45.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-12-16T20:49:45.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-12-16T20:49:45.263 DEBUG:teuthology.orchestra.run.smithi150:> ') 2023-12-16T20:49:45.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:45.402 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec bash -c ' 2023-12-16T20:49:45.402 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2023-12-16T20:49:45.402 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2023-12-16T20:49:45.402 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set dev brx.0 master ceph-brx 2023-12-16T20:49:45.402 DEBUG:teuthology.orchestra.run.smithi150:> ') 2023-12-16T20:49:45.443 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T20:49:45.443 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-12-16T20:49:45.499 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-12-16T20:49:45.499 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T20:49:45.499 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-12-16T20:49:45.553 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-12-16T20:49:45.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:45.553 DEBUG:teuthology.orchestra.run.smithi150:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-16T20:49:45.657 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-16T20:49:45.657 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-12-16T20:49:45.657 INFO:teuthology.orchestra.run.smithi150.stdout:invalid new device string format 2023-12-16T20:49:45.789 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-16T20:49:45.789 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-12-16T20:49:45.789 INFO:teuthology.orchestra.run.smithi150.stdout:invalid new device string format 2023-12-16T20:49:45.789 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: resolved to: "172.21.15.52:3300,172.21.15.150:3300" 2023-12-16T20:49:45.790 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-16T20:49:45.794 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-16T20:49:45.794 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-12-16T20:49:45.794 INFO:teuthology.orchestra.run.smithi150.stdout:invalid new device string format 2023-12-16T20:49:45.794 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: resolved to: "172.21.15.52:3300,172.21.15.150:3300" 2023-12-16T20:49:45.794 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-16T20:49:45.794 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: resolved to: "172.21.15.52:6789,172.21.15.150:6789" 2023-12-16T20:49:45.794 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.52:6789,172.21.15.150:6789:/ 2023-12-16T20:49:45.794 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=cf043a8a-9c53-11ee-95a5-87774f69a715" will pass to kernel 2023-12-16T20:49:45.795 INFO:teuthology.orchestra.run.smithi150.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-12-16T20:49:45.795 INFO:teuthology.orchestra.run.smithi150.stdout: You just mounted a file system that supports labels which does not 2023-12-16T20:49:45.795 INFO:teuthology.orchestra.run.smithi150.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-12-16T20:49:45.795 INFO:teuthology.orchestra.run.smithi150.stdout: applications will generate AVC messages and not be allowed access to 2023-12-16T20:49:45.795 INFO:teuthology.orchestra.run.smithi150.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-12-16T20:49:45.796 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-12-16T20:49:45.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:45.797 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-12-16T20:49:45.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:45.921 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-16T20:49:45.937 INFO:teuthology.orchestra.run.smithi150.stdout:client24309 2023-12-16T20:49:45.938 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-16T20:49:45.938 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T20:49:45.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:45.995 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-16T20:49:46.050 INFO:teuthology.orchestra.run.smithi150.stdout:client24309 2023-12-16T20:49:46.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:46.051 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-16T20:49:46.106 INFO:teuthology.orchestra.run.smithi150.stdout:client24309 2023-12-16T20:49:46.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T20:49:46.107 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715.client24309/status') 2023-12-16T20:49:46.174 INFO:teuthology.orchestra.run.smithi150.stdout:instance: client.24309 (3)192.168.0.1:0/3928881467 2023-12-16T20:49:46.174 INFO:teuthology.orchestra.run.smithi150.stdout:blocklisted: false 2023-12-16T20:49:46.174 INFO:teuthology.orchestra.run.smithi150.stderr:0+1 records in 2023-12-16T20:49:46.174 INFO:teuthology.orchestra.run.smithi150.stderr:0+1 records out 2023-12-16T20:49:46.174 INFO:teuthology.orchestra.run.smithi150.stderr:70 bytes copied, 2.2541e-05 s, 3.1 MB/s 2023-12-16T20:49:46.175 INFO:teuthology.run_tasks:Running task print... 2023-12-16T20:49:46.186 INFO:teuthology.task.print:**** done client 2023-12-16T20:49:46.186 INFO:teuthology.run_tasks:Running task parallel... 2023-12-16T20:49:46.197 INFO:teuthology.task.parallel:starting parallel... 2023-12-16T20:49:46.197 INFO:teuthology.task.parallel:In parallel, running task sequential... 2023-12-16T20:49:46.198 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2023-12-16T20:49:46.198 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi052.front.sepia.ceph.com 2023-12-16T20:49:46.198 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2023-12-16T20:49:46.199 INFO:teuthology.task.parallel:In parallel, running task sequential... 2023-12-16T20:49:46.199 INFO:teuthology.task.sequential:In sequential, running task workunit... 2023-12-16T20:49:46.202 INFO:tasks.workunit:Pulling workunits from ref c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T20:49:46.202 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-16T20:49:46.202 INFO:tasks.workunit:timeout=3h 2023-12-16T20:49:46.202 INFO:tasks.workunit:cleanup=True 2023-12-16T20:49:46.203 DEBUG:teuthology.orchestra.run.smithi052:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-16T20:49:46.223 INFO:teuthology.orchestra.run.smithi052.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-12-16T20:49:46.224 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-12-16T20:49:46.224 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 72h/114d Inode: 1 Links: 2 2023-12-16T20:49:46.224 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T20:49:46.224 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:unlabeled_t:s0 2023-12-16T20:49:46.224 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-12-16T20:49:46.224 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2023-12-16 20:49:23.462906037 +0000 2023-12-16T20:49:46.224 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2023-12-16 20:49:42.436624846 +0000 2023-12-16T20:49:46.224 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2023-12-16 20:49:23.462906037 +0000 2023-12-16T20:49:46.224 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-12-16T20:49:46.224 DEBUG:teuthology.orchestra.run.smithi052:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-12-16T20:49:46.302 DEBUG:teuthology.orchestra.run.smithi150:> stat -- /home/ubuntu/cephtest/mnt.1 2023-12-16T20:49:46.319 INFO:teuthology.orchestra.run.smithi150.stdout: File: /home/ubuntu/cephtest/mnt.1 2023-12-16T20:49:46.319 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2023-12-16T20:49:46.319 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 73h/115d Inode: 1 Links: 3 2023-12-16T20:49:46.319 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T20:49:46.319 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:unlabeled_t:s0 2023-12-16T20:49:46.319 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-12-16T20:49:46.319 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-12-16 20:49:46.296578744 +0000 2023-12-16T20:49:46.319 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-12-16 20:49:46.296578744 +0000 2023-12-16T20:49:46.319 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2023-12-16 20:49:23.462906037 +0000 2023-12-16T20:49:46.319 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2023-12-16T20:49:46.320 DEBUG:teuthology.orchestra.run.smithi150:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2023-12-16T20:49:46.393 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T20:49:46.393 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T20:49:46.409 INFO:tasks.workunit.client.0.smithi052.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-16T20:49:46.448 INFO:tasks.workunit.client.1.smithi150.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2023-12-16T20:49:46.491 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:47.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:46 smithi052 ceph-mon[80761]: pgmap v135: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.9 KiB/s rd, 170 B/s wr, 7 op/s 2023-12-16T20:49:47.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.363+0000 7f8ebd04f700 1 -- 172.21.15.52:0/685896166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8eb80fbaf0 msgr2=0x7f8eb80fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:47.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.363+0000 7f8ebd04f700 1 --2- 172.21.15.52:0/685896166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8eb80fbaf0 0x7f8eb80fbf00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8004940 tx=0x7f8ea8009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:47.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.363+0000 7f8ebd04f700 1 -- 172.21.15.52:0/685896166 shutdown_connections 2023-12-16T20:49:47.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.363+0000 7f8ebd04f700 1 --2- 172.21.15.52:0/685896166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8eb80fcd00 0x7f8eb80fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:47.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.363+0000 7f8ebd04f700 1 --2- 172.21.15.52:0/685896166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8eb80fbaf0 0x7f8eb80fbf00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:47.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.364+0000 7f8ebd04f700 1 -- 172.21.15.52:0/685896166 >> 172.21.15.52:0/685896166 conn(0x7f8eb80f71d0 msgr2=0x7f8eb80f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:47.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.364+0000 7f8ebd04f700 1 -- 172.21.15.52:0/685896166 shutdown_connections 2023-12-16T20:49:47.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.364+0000 7f8ebd04f700 1 -- 172.21.15.52:0/685896166 wait complete. 2023-12-16T20:49:47.366 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.364+0000 7f8ebd04f700 1 Processor -- start 2023-12-16T20:49:47.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.364+0000 7f8ebd04f700 1 -- start start 2023-12-16T20:49:47.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.364+0000 7f8ebd04f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8eb80fbaf0 0x7f8eb810d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:47.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.364+0000 7f8ebd04f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8eb80fcd00 0x7f8eb810dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:47.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.364+0000 7f8ebd04f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb810e0b0 con 0x7f8eb80fcd00 2023-12-16T20:49:47.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.364+0000 7f8ebd04f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb810e950 con 0x7f8eb80fbaf0 2023-12-16T20:49:47.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.365+0000 7f8eb659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8eb80fcd00 0x7f8eb810dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:47.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.365+0000 7f8eb659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8eb80fcd00 0x7f8eb810dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41852/0 (socket says 172.21.15.52:41852) 2023-12-16T20:49:47.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.365+0000 7f8eb659c700 1 -- 172.21.15.52:0/2033054503 learned_addr learned my addr 172.21.15.52:0/2033054503 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:47.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.365+0000 7f8eb6d9d700 1 --2- 172.21.15.52:0/2033054503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8eb80fbaf0 0x7f8eb810d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:47.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.365+0000 7f8eb659c700 1 -- 172.21.15.52:0/2033054503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8eb80fbaf0 msgr2=0x7f8eb810d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:47.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.365+0000 7f8eb659c700 1 --2- 172.21.15.52:0/2033054503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8eb80fbaf0 0x7f8eb810d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:47.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.365+0000 7f8eb659c700 1 -- 172.21.15.52:0/2033054503 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea8005040 con 0x7f8eb80fcd00 2023-12-16T20:49:47.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.365+0000 7f8eb659c700 1 --2- 172.21.15.52:0/2033054503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8eb80fcd00 0x7f8eb810dab0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8eac00c8c0 tx=0x7f8eac0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:47.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.365+0000 7f8e9f7fe700 1 -- 172.21.15.52:0/2033054503 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8eac00a980 con 0x7f8eb80fcd00 2023-12-16T20:49:47.368 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.365+0000 7f8ebd04f700 1 -- 172.21.15.52:0/2033054503 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb810ec30 con 0x7f8eb80fcd00 2023-12-16T20:49:47.368 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.365+0000 7f8e9f7fe700 1 -- 172.21.15.52:0/2033054503 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8eac00ce60 con 0x7f8eb80fcd00 2023-12-16T20:49:47.368 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.365+0000 7f8e9f7fe700 1 -- 172.21.15.52:0/2033054503 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8eac0113a0 con 0x7f8eb80fcd00 2023-12-16T20:49:47.368 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.365+0000 7f8ebd04f700 1 -- 172.21.15.52:0/2033054503 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb810f170 con 0x7f8eb80fcd00 2023-12-16T20:49:47.368 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.366+0000 7f8ebd04f700 1 -- 172.21.15.52:0/2033054503 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb804bb80 con 0x7f8eb80fcd00 2023-12-16T20:49:47.369 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.367+0000 7f8e9f7fe700 1 -- 172.21.15.52:0/2033054503 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f8eac012400 con 0x7f8eb80fcd00 2023-12-16T20:49:47.369 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.367+0000 7f8e9f7fe700 1 --2- 172.21.15.52:0/2033054503 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ea0068f60 0x7f8ea006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:47.369 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.367+0000 7f8e9f7fe700 1 -- 172.21.15.52:0/2033054503 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8eac086f30 con 0x7f8eb80fcd00 2023-12-16T20:49:47.369 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.367+0000 7f8eb6d9d700 1 --2- 172.21.15.52:0/2033054503 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ea0068f60 0x7f8ea006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:47.369 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.368+0000 7f8eb6d9d700 1 --2- 172.21.15.52:0/2033054503 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ea0068f60 0x7f8ea006b410 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800b3b0 tx=0x7f8ea8009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:47.372 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.372+0000 7f8e9f7fe700 1 -- 172.21.15.52:0/2033054503 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8eac056e40 con 0x7f8eb80fcd00 2023-12-16T20:49:47.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:46 smithi150 ceph-mon[88000]: pgmap v135: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.9 KiB/s rd, 170 B/s wr, 7 op/s 2023-12-16T20:49:47.569 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.568+0000 7f8ebd04f700 1 -- 172.21.15.52:0/2033054503 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f8eb8104490 con 0x7f8eb80fcd00 2023-12-16T20:49:47.569 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.568+0000 7f8e9f7fe700 1 -- 172.21.15.52:0/2033054503 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f8eac0569d0 con 0x7f8eb80fcd00 2023-12-16T20:49:47.571 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.570+0000 7f8ebd04f700 1 -- 172.21.15.52:0/2033054503 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ea0068f60 msgr2=0x7f8ea006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:47.571 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.570+0000 7f8ebd04f700 1 --2- 172.21.15.52:0/2033054503 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ea0068f60 0x7f8ea006b410 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800b3b0 tx=0x7f8ea8009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:47.571 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.570+0000 7f8ebd04f700 1 -- 172.21.15.52:0/2033054503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8eb80fcd00 msgr2=0x7f8eb810dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:47.571 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.570+0000 7f8ebd04f700 1 --2- 172.21.15.52:0/2033054503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8eb80fcd00 0x7f8eb810dab0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8eac00c8c0 tx=0x7f8eac0044c0 comp rx=0 tx=0).stop 2023-12-16T20:49:47.571 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.570+0000 7f8ebd04f700 1 -- 172.21.15.52:0/2033054503 shutdown_connections 2023-12-16T20:49:47.571 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.570+0000 7f8ebd04f700 1 --2- 172.21.15.52:0/2033054503 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ea0068f60 0x7f8ea006b410 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:47.571 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.570+0000 7f8ebd04f700 1 --2- 172.21.15.52:0/2033054503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8eb80fbaf0 0x7f8eb810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:47.571 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.570+0000 7f8ebd04f700 1 --2- 172.21.15.52:0/2033054503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8eb80fcd00 0x7f8eb810dab0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:47.571 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.570+0000 7f8ebd04f700 1 -- 172.21.15.52:0/2033054503 >> 172.21.15.52:0/2033054503 conn(0x7f8eb80f71d0 msgr2=0x7f8eb8065c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:47.572 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.570+0000 7f8ebd04f700 1 -- 172.21.15.52:0/2033054503 shutdown_connections 2023-12-16T20:49:47.572 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:47.570+0000 7f8ebd04f700 1 -- 172.21.15.52:0/2033054503 wait complete. 2023-12-16T20:49:47.995 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2023-12-16T20:49:48.302 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:49.172 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.169+0000 7f161ccbb700 1 -- 172.21.15.52:0/1724333037 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16180ffd80 msgr2=0x7f16181021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:49.172 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.169+0000 7f161ccbb700 1 --2- 172.21.15.52:0/1724333037 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16180ffd80 0x7f16181021e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f160c004940 tx=0x7f160c009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:49.172 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.170+0000 7f161ccbb700 1 -- 172.21.15.52:0/1724333037 shutdown_connections 2023-12-16T20:49:49.172 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.170+0000 7f161ccbb700 1 --2- 172.21.15.52:0/1724333037 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16180ffd80 0x7f16181021e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:49.172 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.170+0000 7f161ccbb700 1 --2- 172.21.15.52:0/1724333037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1618066920 0x7f16180ff7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:49.172 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.170+0000 7f161ccbb700 1 -- 172.21.15.52:0/1724333037 >> 172.21.15.52:0/1724333037 conn(0x7f16180f71c0 msgr2=0x7f16180f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:49.173 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.170+0000 7f161ccbb700 1 -- 172.21.15.52:0/1724333037 shutdown_connections 2023-12-16T20:49:49.173 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.170+0000 7f161ccbb700 1 -- 172.21.15.52:0/1724333037 wait complete. 2023-12-16T20:49:49.173 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.170+0000 7f161ccbb700 1 Processor -- start 2023-12-16T20:49:49.173 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.170+0000 7f161ccbb700 1 -- start start 2023-12-16T20:49:49.173 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.170+0000 7f161ccbb700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1618066920 0x7f161810d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:49.173 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.170+0000 7f161ccbb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16180ffd80 0x7f161810da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:49.173 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.170+0000 7f161ccbb700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161810e060 con 0x7f1618066920 2023-12-16T20:49:49.173 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.170+0000 7f161ccbb700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161810e900 con 0x7f16180ffd80 2023-12-16T20:49:49.173 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.171+0000 7f161659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1618066920 0x7f161810d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:49.173 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.171+0000 7f161659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1618066920 0x7f161810d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41870/0 (socket says 172.21.15.52:41870) 2023-12-16T20:49:49.173 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.171+0000 7f161659c700 1 -- 172.21.15.52:0/3660789383 learned_addr learned my addr 172.21.15.52:0/3660789383 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:49.173 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.171+0000 7f1615d9b700 1 --2- 172.21.15.52:0/3660789383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16180ffd80 0x7f161810da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:49.173 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.171+0000 7f161659c700 1 -- 172.21.15.52:0/3660789383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16180ffd80 msgr2=0x7f161810da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:49.174 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.171+0000 7f161659c700 1 --2- 172.21.15.52:0/3660789383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16180ffd80 0x7f161810da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:49.174 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.171+0000 7f161659c700 1 -- 172.21.15.52:0/3660789383 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c005040 con 0x7f1618066920 2023-12-16T20:49:49.174 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.171+0000 7f161659c700 1 --2- 172.21.15.52:0/3660789383 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1618066920 0x7f161810d520 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1608002a50 tx=0x7f16080046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:49.174 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.171+0000 7f1606ffd700 1 -- 172.21.15.52:0/3660789383 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1608007cc0 con 0x7f1618066920 2023-12-16T20:49:49.174 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.171+0000 7f161ccbb700 1 -- 172.21.15.52:0/3660789383 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f161810ebe0 con 0x7f1618066920 2023-12-16T20:49:49.174 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.171+0000 7f161ccbb700 1 -- 172.21.15.52:0/3660789383 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f161810f120 con 0x7f1618066920 2023-12-16T20:49:49.174 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.172+0000 7f1606ffd700 1 -- 172.21.15.52:0/3660789383 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f160800f040 con 0x7f1618066920 2023-12-16T20:49:49.174 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.172+0000 7f1606ffd700 1 -- 172.21.15.52:0/3660789383 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f160800d9c0 con 0x7f1618066920 2023-12-16T20:49:49.175 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.172+0000 7f161ccbb700 1 -- 172.21.15.52:0/3660789383 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16181075d0 con 0x7f1618066920 2023-12-16T20:49:49.175 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.173+0000 7f1606ffd700 1 -- 172.21.15.52:0/3660789383 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1608009070 con 0x7f1618066920 2023-12-16T20:49:49.175 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.173+0000 7f1606ffd700 1 --2- 172.21.15.52:0/3660789383 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1600068f60 0x7f160006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:49.175 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.174+0000 7f1606ffd700 1 -- 172.21.15.52:0/3660789383 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1608086a60 con 0x7f1618066920 2023-12-16T20:49:49.175 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.174+0000 7f1615d9b700 1 --2- 172.21.15.52:0/3660789383 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1600068f60 0x7f160006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:49.175 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.174+0000 7f1615d9b700 1 --2- 172.21.15.52:0/3660789383 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1600068f60 0x7f160006b410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f160c004890 tx=0x7f160c009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:49.179 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.178+0000 7f1606ffd700 1 -- 172.21.15.52:0/3660789383 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1608056970 con 0x7f1618066920 2023-12-16T20:49:49.197 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:48 smithi052 ceph-mon[80761]: pgmap v136: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 682 B/s wr, 8 op/s 2023-12-16T20:49:49.197 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:48 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:49.197 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:48 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:49:49.377 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.376+0000 7f161ccbb700 1 -- 172.21.15.52:0/3660789383 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f161804bb80 con 0x7f1618066920 2023-12-16T20:49:49.377 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.377+0000 7f1606ffd700 1 -- 172.21.15.52:0/3660789383 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f1608056500 con 0x7f1618066920 2023-12-16T20:49:49.379 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.379+0000 7f161ccbb700 1 -- 172.21.15.52:0/3660789383 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1600068f60 msgr2=0x7f160006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:49.379 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.379+0000 7f161ccbb700 1 --2- 172.21.15.52:0/3660789383 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1600068f60 0x7f160006b410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f160c004890 tx=0x7f160c009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:49.379 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.379+0000 7f161ccbb700 1 -- 172.21.15.52:0/3660789383 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1618066920 msgr2=0x7f161810d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:49.380 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.379+0000 7f161ccbb700 1 --2- 172.21.15.52:0/3660789383 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1618066920 0x7f161810d520 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1608002a50 tx=0x7f16080046b0 comp rx=0 tx=0).stop 2023-12-16T20:49:49.380 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.379+0000 7f161ccbb700 1 -- 172.21.15.52:0/3660789383 shutdown_connections 2023-12-16T20:49:49.380 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.379+0000 7f161ccbb700 1 --2- 172.21.15.52:0/3660789383 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1600068f60 0x7f160006b410 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:49.380 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.379+0000 7f161ccbb700 1 --2- 172.21.15.52:0/3660789383 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1618066920 0x7f161810d520 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:49.380 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.379+0000 7f161ccbb700 1 --2- 172.21.15.52:0/3660789383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16180ffd80 0x7f161810da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:49.380 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.379+0000 7f161ccbb700 1 -- 172.21.15.52:0/3660789383 >> 172.21.15.52:0/3660789383 conn(0x7f16180f71c0 msgr2=0x7f16180f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:49.380 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.379+0000 7f161ccbb700 1 -- 172.21.15.52:0/3660789383 shutdown_connections 2023-12-16T20:49:49.380 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:49.379+0000 7f161ccbb700 1 -- 172.21.15.52:0/3660789383 wait complete. 2023-12-16T20:49:49.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:48 smithi150 ceph-mon[88000]: pgmap v136: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 682 B/s wr, 8 op/s 2023-12-16T20:49:49.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:48 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:49:49.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:48 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:49:49.783 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph config set global log_to_journald false --force' 2023-12-16T20:49:50.050 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:51.009 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.006+0000 7fada51f4700 1 -- 172.21.15.52:0/1100972215 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fada00fdee0 msgr2=0x7fada00fe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:51.009 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.006+0000 7fada51f4700 1 --2- 172.21.15.52:0/1100972215 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fada00fdee0 0x7fada00fe330 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fad90004940 tx=0x7fad90009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:51.009 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.006+0000 7fada51f4700 1 -- 172.21.15.52:0/1100972215 shutdown_connections 2023-12-16T20:49:51.009 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.006+0000 7fada51f4700 1 --2- 172.21.15.52:0/1100972215 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fada00fdee0 0x7fada00fe330 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:51.009 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.006+0000 7fada51f4700 1 --2- 172.21.15.52:0/1100972215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fada00fae50 0x7fada00fb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:51.010 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.006+0000 7fada51f4700 1 -- 172.21.15.52:0/1100972215 >> 172.21.15.52:0/1100972215 conn(0x7fada00f68b0 msgr2=0x7fada00f8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:51.010 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.007+0000 7fada51f4700 1 -- 172.21.15.52:0/1100972215 shutdown_connections 2023-12-16T20:49:51.010 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.007+0000 7fada51f4700 1 -- 172.21.15.52:0/1100972215 wait complete. 2023-12-16T20:49:51.010 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.007+0000 7fada51f4700 1 Processor -- start 2023-12-16T20:49:51.010 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.007+0000 7fada51f4700 1 -- start start 2023-12-16T20:49:51.010 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.007+0000 7fada51f4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fada00fae50 0x7fada010d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:51.010 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.007+0000 7fada51f4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fada00fdee0 0x7fada010d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:51.010 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.007+0000 7fada51f4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada010e080 con 0x7fada00fae50 2023-12-16T20:49:51.010 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.007+0000 7fada51f4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada010e8c0 con 0x7fada00fdee0 2023-12-16T20:49:51.010 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.008+0000 7fad9ed9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fada00fae50 0x7fada010d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:51.010 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.008+0000 7fad9ed9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fada00fae50 0x7fada010d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41880/0 (socket says 172.21.15.52:41880) 2023-12-16T20:49:51.010 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.008+0000 7fad9ed9d700 1 -- 172.21.15.52:0/1951578515 learned_addr learned my addr 172.21.15.52:0/1951578515 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:51.010 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.008+0000 7fad9e59c700 1 --2- 172.21.15.52:0/1951578515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fada00fdee0 0x7fada010d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:51.011 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.008+0000 7fad9ed9d700 1 -- 172.21.15.52:0/1951578515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fada00fdee0 msgr2=0x7fada010d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:51.011 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.008+0000 7fad9ed9d700 1 --2- 172.21.15.52:0/1951578515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fada00fdee0 0x7fada010d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:51.011 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.008+0000 7fad9ed9d700 1 -- 172.21.15.52:0/1951578515 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad90005040 con 0x7fada00fae50 2023-12-16T20:49:51.011 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.008+0000 7fad9ed9d700 1 --2- 172.21.15.52:0/1951578515 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fada00fae50 0x7fada010d4b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fad9400b790 tx=0x7fad94009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:51.011 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.008+0000 7fad877fe700 1 -- 172.21.15.52:0/1951578515 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad9400f6f0 con 0x7fada00fae50 2023-12-16T20:49:51.011 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.008+0000 7fada51f4700 1 -- 172.21.15.52:0/1951578515 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada010eba0 con 0x7fada00fae50 2023-12-16T20:49:51.011 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.008+0000 7fad877fe700 1 -- 172.21.15.52:0/1951578515 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad9400fd00 con 0x7fada00fae50 2023-12-16T20:49:51.011 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.008+0000 7fad877fe700 1 -- 172.21.15.52:0/1951578515 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad94011e90 con 0x7fada00fae50 2023-12-16T20:49:51.011 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.008+0000 7fada51f4700 1 -- 172.21.15.52:0/1951578515 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada010f160 con 0x7fada00fae50 2023-12-16T20:49:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.009+0000 7fada51f4700 1 -- 172.21.15.52:0/1951578515 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada004bb80 con 0x7fada00fae50 2023-12-16T20:49:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.010+0000 7fad877fe700 1 -- 172.21.15.52:0/1951578515 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fad9400f850 con 0x7fada00fae50 2023-12-16T20:49:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.010+0000 7fad877fe700 1 --2- 172.21.15.52:0/1951578515 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fad88068f70 0x7fad8806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.010+0000 7fad877fe700 1 -- 172.21.15.52:0/1951578515 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fad94086bc0 con 0x7fada00fae50 2023-12-16T20:49:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.010+0000 7fad9e59c700 1 --2- 172.21.15.52:0/1951578515 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fad88068f70 0x7fad8806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.011+0000 7fad9e59c700 1 --2- 172.21.15.52:0/1951578515 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fad88068f70 0x7fad8806b420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fad9000b3b0 tx=0x7fad90009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:51.016 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.015+0000 7fad877fe700 1 -- 172.21.15.52:0/1951578515 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fad94056ad0 con 0x7fada00fae50 2023-12-16T20:49:51.210 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.210+0000 7fada51f4700 1 -- 172.21.15.52:0/1951578515 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fada010f5e0 con 0x7fada00fae50 2023-12-16T20:49:51.211 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:50 smithi052 ceph-mon[80761]: pgmap v137: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 6 op/s 2023-12-16T20:49:51.211 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.210+0000 7fad877fe700 1 -- 172.21.15.52:0/1951578515 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7fad94056660 con 0x7fada00fae50 2023-12-16T20:49:51.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.212+0000 7fada51f4700 1 -- 172.21.15.52:0/1951578515 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fad88068f70 msgr2=0x7fad8806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:51.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.212+0000 7fada51f4700 1 --2- 172.21.15.52:0/1951578515 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fad88068f70 0x7fad8806b420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fad9000b3b0 tx=0x7fad90009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:51.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.212+0000 7fada51f4700 1 -- 172.21.15.52:0/1951578515 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fada00fae50 msgr2=0x7fada010d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:51.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.212+0000 7fada51f4700 1 --2- 172.21.15.52:0/1951578515 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fada00fae50 0x7fada010d4b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fad9400b790 tx=0x7fad94009220 comp rx=0 tx=0).stop 2023-12-16T20:49:51.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.212+0000 7fada51f4700 1 -- 172.21.15.52:0/1951578515 shutdown_connections 2023-12-16T20:49:51.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.212+0000 7fada51f4700 1 --2- 172.21.15.52:0/1951578515 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fad88068f70 0x7fad8806b420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:51.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.212+0000 7fada51f4700 1 --2- 172.21.15.52:0/1951578515 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fada00fae50 0x7fada010d4b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:51.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.212+0000 7fada51f4700 1 --2- 172.21.15.52:0/1951578515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fada00fdee0 0x7fada010d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:51.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.212+0000 7fada51f4700 1 -- 172.21.15.52:0/1951578515 >> 172.21.15.52:0/1951578515 conn(0x7fada00f68b0 msgr2=0x7fada0101110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:51.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.212+0000 7fada51f4700 1 -- 172.21.15.52:0/1951578515 shutdown_connections 2023-12-16T20:49:51.214 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:51.212+0000 7fada51f4700 1 -- 172.21.15.52:0/1951578515 wait complete. 2023-12-16T20:49:51.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:50 smithi150 ceph-mon[88000]: pgmap v137: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 6 op/s 2023-12-16T20:49:51.667 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2023-12-16T20:49:51.935 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:49:52.808 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.807+0000 7f43bbae3700 1 -- 172.21.15.52:0/1402942288 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43b40fcd00 msgr2=0x7f43b40fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:52.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.807+0000 7f43bbae3700 1 --2- 172.21.15.52:0/1402942288 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43b40fcd00 0x7f43b40fd150 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f43a4004940 tx=0x7f43a4009e30 comp rx=0 tx=0).stop 2023-12-16T20:49:52.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.807+0000 7f43bbae3700 1 -- 172.21.15.52:0/1402942288 shutdown_connections 2023-12-16T20:49:52.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.807+0000 7f43bbae3700 1 --2- 172.21.15.52:0/1402942288 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43b40fcd00 0x7f43b40fd150 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:52.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.807+0000 7f43bbae3700 1 --2- 172.21.15.52:0/1402942288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b40fbaf0 0x7f43b40fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:52.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.807+0000 7f43bbae3700 1 -- 172.21.15.52:0/1402942288 >> 172.21.15.52:0/1402942288 conn(0x7f43b40f7190 msgr2=0x7f43b40f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:49:52.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.807+0000 7f43bbae3700 1 -- 172.21.15.52:0/1402942288 shutdown_connections 2023-12-16T20:49:52.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.807+0000 7f43bbae3700 1 -- 172.21.15.52:0/1402942288 wait complete. 2023-12-16T20:49:52.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.808+0000 7f43bbae3700 1 Processor -- start 2023-12-16T20:49:52.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.808+0000 7f43bbae3700 1 -- start start 2023-12-16T20:49:52.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.808+0000 7f43bbae3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b40fbaf0 0x7f43b40ffbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:52.809 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.808+0000 7f43bbae3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43b40fcd00 0x7f43b40fe040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:52.810 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.808+0000 7f43bbae3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b41111b0 con 0x7f43b40fcd00 2023-12-16T20:49:52.810 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.808+0000 7f43bbae3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b4111320 con 0x7f43b40fbaf0 2023-12-16T20:49:52.810 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.808+0000 7f43b907e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43b40fcd00 0x7f43b40fe040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:52.810 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.808+0000 7f43b907e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43b40fcd00 0x7f43b40fe040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60418/0 (socket says 172.21.15.52:60418) 2023-12-16T20:49:52.810 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.808+0000 7f43b907e700 1 -- 172.21.15.52:0/2124342112 learned_addr learned my addr 172.21.15.52:0/2124342112 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:49:52.810 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.808+0000 7f43b987f700 1 --2- 172.21.15.52:0/2124342112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b40fbaf0 0x7f43b40ffbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:52.811 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.808+0000 7f43b907e700 1 -- 172.21.15.52:0/2124342112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b40fbaf0 msgr2=0x7f43b40ffbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:49:52.811 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.808+0000 7f43b907e700 1 --2- 172.21.15.52:0/2124342112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b40fbaf0 0x7f43b40ffbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:49:52.811 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.808+0000 7f43b907e700 1 -- 172.21.15.52:0/2124342112 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a4005040 con 0x7f43b40fcd00 2023-12-16T20:49:52.811 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.809+0000 7f43b907e700 1 --2- 172.21.15.52:0/2124342112 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43b40fcd00 0x7f43b40fe040 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f43a4009380 tx=0x7f43a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:52.811 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.809+0000 7f43aa7fc700 1 -- 172.21.15.52:0/2124342112 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43a400e040 con 0x7f43b40fcd00 2023-12-16T20:49:52.811 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.809+0000 7f43aa7fc700 1 -- 172.21.15.52:0/2124342112 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43a400bd20 con 0x7f43b40fcd00 2023-12-16T20:49:52.811 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.809+0000 7f43bbae3700 1 -- 172.21.15.52:0/2124342112 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b40fe640 con 0x7f43b40fcd00 2023-12-16T20:49:52.811 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.809+0000 7f43aa7fc700 1 -- 172.21.15.52:0/2124342112 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43a401c630 con 0x7f43b40fcd00 2023-12-16T20:49:52.812 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.809+0000 7f43bbae3700 1 -- 172.21.15.52:0/2124342112 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b40feb80 con 0x7f43b40fcd00 2023-12-16T20:49:52.812 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.810+0000 7f43bbae3700 1 -- 172.21.15.52:0/2124342112 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b404bb80 con 0x7f43b40fcd00 2023-12-16T20:49:52.812 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.810+0000 7f43aa7fc700 1 -- 172.21.15.52:0/2124342112 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f43a4013420 con 0x7f43b40fcd00 2023-12-16T20:49:52.812 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.811+0000 7f43aa7fc700 1 --2- 172.21.15.52:0/2124342112 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f43a0068f60 0x7f43a006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:49:52.812 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.811+0000 7f43aa7fc700 1 -- 172.21.15.52:0/2124342112 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f43a4088180 con 0x7f43b40fcd00 2023-12-16T20:49:52.813 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.811+0000 7f43b987f700 1 --2- 172.21.15.52:0/2124342112 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f43a0068f60 0x7f43a006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:49:52.813 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.811+0000 7f43b987f700 1 --2- 172.21.15.52:0/2124342112 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f43a0068f60 0x7f43a006b410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f43b40ff950 tx=0x7f43b000b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:49:52.816 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:52.815+0000 7f43aa7fc700 1 -- 172.21.15.52:0/2124342112 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f43a4058090 con 0x7f43b40fcd00 2023-12-16T20:49:53.038 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:49:53.038+0000 7f43bbae3700 1 -- 172.21.15.52:0/2124342112 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f43b4065a80 con 0x7f43a0068f60 2023-12-16T20:49:53.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:52 smithi052 ceph-mon[80761]: pgmap v138: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 853 B/s wr, 4 op/s 2023-12-16T20:49:53.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:52 smithi150 ceph-mon[88000]: pgmap v138: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 853 B/s wr, 4 op/s 2023-12-16T20:49:54.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:53 smithi052 ceph-mon[80761]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:49:54.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:53 smithi150 ceph-mon[88000]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:49:55.187 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:54 smithi150 ceph-mon[88000]: pgmap v139: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 5 op/s 2023-12-16T20:49:55.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:54 smithi052 ceph-mon[80761]: pgmap v139: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 5 op/s 2023-12-16T20:49:57.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:56 smithi052 ceph-mon[80761]: pgmap v140: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 4 op/s 2023-12-16T20:49:57.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:56 smithi150 ceph-mon[88000]: pgmap v140: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 4 op/s 2023-12-16T20:49:58.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:49:58 smithi052 ceph-mon[80761]: pgmap v141: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2023-12-16T20:49:58.937 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:49:58 smithi150 ceph-mon[88000]: pgmap v141: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2023-12-16T20:50:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:00 smithi052 ceph-mon[80761]: pgmap v142: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2023-12-16T20:50:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:00 smithi052 ceph-mon[80761]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2023-12-16T20:50:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:00 smithi052 ceph-mon[80761]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2023-12-16T20:50:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:00 smithi052 ceph-mon[80761]: fs cephfs has deprecated feature inline_data enabled. 2023-12-16T20:50:01.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:00 smithi150 ceph-mon[88000]: pgmap v142: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2023-12-16T20:50:01.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:00 smithi150 ceph-mon[88000]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2023-12-16T20:50:01.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:00 smithi150 ceph-mon[88000]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2023-12-16T20:50:01.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:00 smithi150 ceph-mon[88000]: fs cephfs has deprecated feature inline_data enabled. 2023-12-16T20:50:03.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:02 smithi052 ceph-mon[80761]: pgmap v143: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2023-12-16T20:50:03.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:02 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:50:03.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:02 smithi150 ceph-mon[88000]: pgmap v143: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2023-12-16T20:50:03.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:02 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:50:05.239 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:04 smithi150 ceph-mon[88000]: pgmap v144: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2023-12-16T20:50:05.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:04 smithi052 ceph-mon[80761]: pgmap v144: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2023-12-16T20:50:07.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:06 smithi150 ceph-mon[88000]: pgmap v145: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2023-12-16T20:50:07.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:06 smithi052 ceph-mon[80761]: pgmap v145: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2023-12-16T20:50:08.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:08 smithi052 ceph-mon[80761]: pgmap v146: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2023-12-16T20:50:08.937 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:08 smithi150 ceph-mon[88000]: pgmap v146: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2023-12-16T20:50:11.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:10 smithi052 ceph-mon[80761]: pgmap v147: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2023-12-16T20:50:11.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:10 smithi150 ceph-mon[88000]: pgmap v147: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2023-12-16T20:50:13.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:12 smithi052 ceph-mon[80761]: pgmap v148: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2023-12-16T20:50:13.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:12 smithi150 ceph-mon[88000]: pgmap v148: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2023-12-16T20:50:15.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:14 smithi052 ceph-mon[80761]: pgmap v149: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2023-12-16T20:50:15.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:14 smithi150 ceph-mon[88000]: pgmap v149: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2023-12-16T20:50:17.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:16 smithi150 ceph-mon[88000]: pgmap v150: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2023-12-16T20:50:17.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:16 smithi052 ceph-mon[80761]: pgmap v150: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2023-12-16T20:50:18.697 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:18 smithi052 ceph-mon[80761]: pgmap v151: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2023-12-16T20:50:18.697 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:18 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:50:18.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:18 smithi150 ceph-mon[88000]: pgmap v151: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2023-12-16T20:50:18.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:18 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:50:19.309 INFO:teuthology.orchestra.run.smithi052.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T20:50:19.309 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:19.307+0000 7f43aa7fc700 1 -- 172.21.15.52:0/2124342112 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f43b4065a80 con 0x7f43a0068f60 2023-12-16T20:50:19.311 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:19.310+0000 7f43bbae3700 1 -- 172.21.15.52:0/2124342112 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f43a0068f60 msgr2=0x7f43a006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:19.311 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:19.310+0000 7f43bbae3700 1 --2- 172.21.15.52:0/2124342112 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f43a0068f60 0x7f43a006b410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f43b40ff950 tx=0x7f43b000b040 comp rx=0 tx=0).stop 2023-12-16T20:50:19.311 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:19.310+0000 7f43bbae3700 1 -- 172.21.15.52:0/2124342112 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43b40fcd00 msgr2=0x7f43b40fe040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:19.311 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:19.310+0000 7f43bbae3700 1 --2- 172.21.15.52:0/2124342112 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43b40fcd00 0x7f43b40fe040 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f43a4009380 tx=0x7f43a4009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:19.311 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:19.311+0000 7f43bbae3700 1 -- 172.21.15.52:0/2124342112 shutdown_connections 2023-12-16T20:50:19.311 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:19.311+0000 7f43bbae3700 1 --2- 172.21.15.52:0/2124342112 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f43a0068f60 0x7f43a006b410 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:19.311 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:19.311+0000 7f43bbae3700 1 --2- 172.21.15.52:0/2124342112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b40fbaf0 0x7f43b40ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:19.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:19.311+0000 7f43bbae3700 1 --2- 172.21.15.52:0/2124342112 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f43b40fcd00 0x7f43b40fe040 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:19.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:19.311+0000 7f43bbae3700 1 -- 172.21.15.52:0/2124342112 >> 172.21.15.52:0/2124342112 conn(0x7f43b40f7190 msgr2=0x7f43b4065370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:19.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:19.311+0000 7f43bbae3700 1 -- 172.21.15.52:0/2124342112 shutdown_connections 2023-12-16T20:50:19.312 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:19.311+0000 7f43bbae3700 1 -- 172.21.15.52:0/2124342112 wait complete. 2023-12-16T20:50:19.710 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2023-12-16T20:50:20.044 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:50:20.434 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:20 smithi052 ceph-mon[80761]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T20:50:20.434 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:20 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:50:20.434 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:20 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:50:20.434 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:20 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:50:20.434 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:20 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:50:20.434 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:20 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:50:20.434 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:20 smithi052 ceph-mon[80761]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T20:50:20.434 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:20 smithi052 ceph-mon[80761]: pgmap v152: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2023-12-16T20:50:20.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:20 smithi150 ceph-mon[88000]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T20:50:20.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:20 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:50:20.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:20 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:50:20.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:20 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:50:20.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:20 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T20:50:20.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:20 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:50:20.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:20 smithi150 ceph-mon[88000]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T20:50:20.688 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:20 smithi150 ceph-mon[88000]: pgmap v152: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2023-12-16T20:50:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.717+0000 7f1a79477700 1 -- 172.21.15.52:0/28862445 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a740fcd00 msgr2=0x7f1a740fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.717+0000 7f1a79477700 1 --2- 172.21.15.52:0/28862445 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a740fcd00 0x7f1a740fd150 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1a68004940 tx=0x7f1a68009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.717+0000 7f1a79477700 1 -- 172.21.15.52:0/28862445 shutdown_connections 2023-12-16T20:50:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.717+0000 7f1a79477700 1 --2- 172.21.15.52:0/28862445 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a740fcd00 0x7f1a740fd150 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.717+0000 7f1a79477700 1 --2- 172.21.15.52:0/28862445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a740fbaf0 0x7f1a740fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.717+0000 7f1a79477700 1 -- 172.21.15.52:0/28862445 >> 172.21.15.52:0/28862445 conn(0x7f1a740f7190 msgr2=0x7f1a740f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.717+0000 7f1a79477700 1 -- 172.21.15.52:0/28862445 shutdown_connections 2023-12-16T20:50:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.717+0000 7f1a79477700 1 -- 172.21.15.52:0/28862445 wait complete. 2023-12-16T20:50:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.718+0000 7f1a79477700 1 Processor -- start 2023-12-16T20:50:21.722 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.718+0000 7f1a79477700 1 -- start start 2023-12-16T20:50:21.722 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.718+0000 7f1a79477700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a740fbaf0 0x7f1a741174a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:21.722 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.718+0000 7f1a79477700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a741179e0 0x7f1a74115f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:21.722 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.719+0000 7f1a727fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a741179e0 0x7f1a74115f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:21.722 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.719+0000 7f1a727fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a741179e0 0x7f1a74115f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36638/0 (socket says 172.21.15.52:36638) 2023-12-16T20:50:21.722 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.719+0000 7f1a727fc700 1 -- 172.21.15.52:0/3215874394 learned_addr learned my addr 172.21.15.52:0/3215874394 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:50:21.722 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.719+0000 7f1a79477700 1 -- 172.21.15.52:0/3215874394 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a74117f60 con 0x7f1a741179e0 2023-12-16T20:50:21.722 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.719+0000 7f1a79477700 1 -- 172.21.15.52:0/3215874394 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a741180d0 con 0x7f1a740fbaf0 2023-12-16T20:50:21.722 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.719+0000 7f1a72ffd700 1 --2- 172.21.15.52:0/3215874394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a740fbaf0 0x7f1a741174a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:21.722 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.719+0000 7f1a727fc700 1 -- 172.21.15.52:0/3215874394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a740fbaf0 msgr2=0x7f1a741174a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:21.722 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.719+0000 7f1a727fc700 1 --2- 172.21.15.52:0/3215874394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a740fbaf0 0x7f1a741174a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:21.722 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.719+0000 7f1a727fc700 1 -- 172.21.15.52:0/3215874394 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a68005040 con 0x7f1a741179e0 2023-12-16T20:50:21.722 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.719+0000 7f1a727fc700 1 --2- 172.21.15.52:0/3215874394 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a741179e0 0x7f1a74115f20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1a6800b0d0 tx=0x7f1a68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:21.723 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.720+0000 7f1a5b7fe700 1 -- 172.21.15.52:0/3215874394 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a6800e030 con 0x7f1a741179e0 2023-12-16T20:50:21.723 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.720+0000 7f1a5b7fe700 1 -- 172.21.15.52:0/3215874394 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a68013c60 con 0x7f1a741179e0 2023-12-16T20:50:21.723 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.720+0000 7f1a5b7fe700 1 -- 172.21.15.52:0/3215874394 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a6801f070 con 0x7f1a741179e0 2023-12-16T20:50:21.723 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.720+0000 7f1a79477700 1 -- 172.21.15.52:0/3215874394 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a74116520 con 0x7f1a741179e0 2023-12-16T20:50:21.723 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.720+0000 7f1a79477700 1 -- 172.21.15.52:0/3215874394 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a74116a60 con 0x7f1a741179e0 2023-12-16T20:50:21.724 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.722+0000 7f1a5b7fe700 1 -- 172.21.15.52:0/3215874394 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1a6801a570 con 0x7f1a741179e0 2023-12-16T20:50:21.724 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.722+0000 7f1a79477700 1 -- 172.21.15.52:0/3215874394 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a74069dd0 con 0x7f1a741179e0 2023-12-16T20:50:21.724 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.723+0000 7f1a5b7fe700 1 --2- 172.21.15.52:0/3215874394 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1a5c068f60 0x7f1a5c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:21.724 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.723+0000 7f1a5b7fe700 1 -- 172.21.15.52:0/3215874394 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1a6808c3d0 con 0x7f1a741179e0 2023-12-16T20:50:21.724 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.724+0000 7f1a72ffd700 1 --2- 172.21.15.52:0/3215874394 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1a5c068f60 0x7f1a5c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:21.725 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.724+0000 7f1a72ffd700 1 --2- 172.21.15.52:0/3215874394 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1a5c068f60 0x7f1a5c06b410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1a740fcb60 tx=0x7f1a6400b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:21.731 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:21.731+0000 7f1a5b7fe700 1 -- 172.21.15.52:0/3215874394 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1a6805c380 con 0x7f1a741179e0 2023-12-16T20:50:22.160 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.160+0000 7f1a79477700 1 -- 172.21.15.52:0/3215874394 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a741045f0 con 0x7f1a5c068f60 2023-12-16T20:50:22.165 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.161+0000 7f1a5b7fe700 1 -- 172.21.15.52:0/3215874394 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f1a741045f0 con 0x7f1a5c068f60 2023-12-16T20:50:22.165 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.164+0000 7f1a79477700 1 -- 172.21.15.52:0/3215874394 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1a5c068f60 msgr2=0x7f1a5c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:22.165 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.164+0000 7f1a79477700 1 --2- 172.21.15.52:0/3215874394 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1a5c068f60 0x7f1a5c06b410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1a740fcb60 tx=0x7f1a6400b040 comp rx=0 tx=0).stop 2023-12-16T20:50:22.165 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.164+0000 7f1a79477700 1 -- 172.21.15.52:0/3215874394 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a741179e0 msgr2=0x7f1a74115f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:22.165 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.164+0000 7f1a79477700 1 --2- 172.21.15.52:0/3215874394 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a741179e0 0x7f1a74115f20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1a6800b0d0 tx=0x7f1a68009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:22.165 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.164+0000 7f1a79477700 1 -- 172.21.15.52:0/3215874394 shutdown_connections 2023-12-16T20:50:22.166 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.164+0000 7f1a79477700 1 --2- 172.21.15.52:0/3215874394 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f1a5c068f60 0x7f1a5c06b410 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:22.166 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.164+0000 7f1a79477700 1 --2- 172.21.15.52:0/3215874394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a740fbaf0 0x7f1a741174a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:22.166 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.164+0000 7f1a79477700 1 --2- 172.21.15.52:0/3215874394 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a741179e0 0x7f1a74115f20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:22.166 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.164+0000 7f1a79477700 1 -- 172.21.15.52:0/3215874394 >> 172.21.15.52:0/3215874394 conn(0x7f1a740f7190 msgr2=0x7f1a740f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:22.166 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.165+0000 7f1a79477700 1 -- 172.21.15.52:0/3215874394 shutdown_connections 2023-12-16T20:50:22.166 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.165+0000 7f1a79477700 1 -- 172.21.15.52:0/3215874394 wait complete. 2023-12-16T20:50:22.180 INFO:teuthology.orchestra.run.smithi052.stdout:true 2023-12-16T20:50:22.303 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.302+0000 7fe76a3cf700 1 -- 172.21.15.52:0/1185984757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7640fbaf0 msgr2=0x7fe7640fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:22.303 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.302+0000 7fe76a3cf700 1 --2- 172.21.15.52:0/1185984757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7640fbaf0 0x7fe7640fbf00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe754004880 tx=0x7fe754009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:22.303 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.302+0000 7fe76a3cf700 1 -- 172.21.15.52:0/1185984757 shutdown_connections 2023-12-16T20:50:22.303 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.302+0000 7fe76a3cf700 1 --2- 172.21.15.52:0/1185984757 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7640fcd00 0x7fe7640fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:22.304 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.302+0000 7fe76a3cf700 1 --2- 172.21.15.52:0/1185984757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7640fbaf0 0x7fe7640fbf00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:22.304 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.302+0000 7fe76a3cf700 1 -- 172.21.15.52:0/1185984757 >> 172.21.15.52:0/1185984757 conn(0x7fe7640f7190 msgr2=0x7fe7640f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:22.304 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.303+0000 7fe76a3cf700 1 -- 172.21.15.52:0/1185984757 shutdown_connections 2023-12-16T20:50:22.304 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.303+0000 7fe76a3cf700 1 -- 172.21.15.52:0/1185984757 wait complete. 2023-12-16T20:50:22.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.303+0000 7fe76a3cf700 1 Processor -- start 2023-12-16T20:50:22.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.303+0000 7fe76a3cf700 1 -- start start 2023-12-16T20:50:22.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.304+0000 7fe76a3cf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7640fbaf0 0x7fe764110210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:22.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.304+0000 7fe76a3cf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7640fcd00 0x7fe764112760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:22.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.304+0000 7fe76a3cf700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe764066110 con 0x7fe7640fbaf0 2023-12-16T20:50:22.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.304+0000 7fe76a3cf700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe764066250 con 0x7fe7640fcd00 2023-12-16T20:50:22.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.304+0000 7fe763fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7640fbaf0 0x7fe764110210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:22.305 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.304+0000 7fe7637fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7640fcd00 0x7fe764112760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:22.306 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.304+0000 7fe763fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7640fbaf0 0x7fe764110210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36666/0 (socket says 172.21.15.52:36666) 2023-12-16T20:50:22.306 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.304+0000 7fe7637fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7640fcd00 0x7fe764112760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.52:49656/0 (socket says 172.21.15.52:49656) 2023-12-16T20:50:22.306 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.304+0000 7fe763fff700 1 -- 172.21.15.52:0/1147396965 learned_addr learned my addr 172.21.15.52:0/1147396965 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:50:22.306 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.304+0000 7fe7637fe700 1 -- 172.21.15.52:0/1147396965 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7640fbaf0 msgr2=0x7fe764110210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:22.306 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.304+0000 7fe7637fe700 1 --2- 172.21.15.52:0/1147396965 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7640fbaf0 0x7fe764110210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:22.306 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.304+0000 7fe7637fe700 1 -- 172.21.15.52:0/1147396965 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe758005040 con 0x7fe7640fcd00 2023-12-16T20:50:22.306 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.305+0000 7fe7637fe700 1 --2- 172.21.15.52:0/1147396965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7640fcd00 0x7fe764112760 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe7580096d0 tx=0x7fe758009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:22.307 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.305+0000 7fe760ff9700 1 -- 172.21.15.52:0/1147396965 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe758017070 con 0x7fe7640fcd00 2023-12-16T20:50:22.307 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.305+0000 7fe76a3cf700 1 -- 172.21.15.52:0/1147396965 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe754005040 con 0x7fe7640fcd00 2023-12-16T20:50:22.307 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.305+0000 7fe760ff9700 1 -- 172.21.15.52:0/1147396965 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe75800f040 con 0x7fe7640fcd00 2023-12-16T20:50:22.307 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.305+0000 7fe760ff9700 1 -- 172.21.15.52:0/1147396965 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe758013ac0 con 0x7fe7640fcd00 2023-12-16T20:50:22.307 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.305+0000 7fe76a3cf700 1 -- 172.21.15.52:0/1147396965 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76410c2a0 con 0x7fe7640fcd00 2023-12-16T20:50:22.307 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.306+0000 7fe76a3cf700 1 -- 172.21.15.52:0/1147396965 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76404bb80 con 0x7fe7640fcd00 2023-12-16T20:50:22.308 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.307+0000 7fe760ff9700 1 -- 172.21.15.52:0/1147396965 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe75800e840 con 0x7fe7640fcd00 2023-12-16T20:50:22.308 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.307+0000 7fe760ff9700 1 --2- 172.21.15.52:0/1147396965 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe74c068f60 0x7fe74c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:22.308 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.307+0000 7fe760ff9700 1 -- 172.21.15.52:0/1147396965 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe758086ba0 con 0x7fe7640fcd00 2023-12-16T20:50:22.308 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.307+0000 7fe763fff700 1 --2- 172.21.15.52:0/1147396965 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe74c068f60 0x7fe74c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:22.308 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.308+0000 7fe763fff700 1 --2- 172.21.15.52:0/1147396965 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe74c068f60 0x7fe74c06b410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe7540135d0 tx=0x7fe754009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:22.316 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.315+0000 7fe760ff9700 1 -- 172.21.15.52:0/1147396965 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe758052f00 con 0x7fe7640fcd00 2023-12-16T20:50:22.743 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.742+0000 7fe76a3cf700 1 -- 172.21.15.52:0/1147396965 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe764101650 con 0x7fe74c068f60 2023-12-16T20:50:22.744 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.744+0000 7fe760ff9700 1 -- 172.21.15.52:0/1147396965 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fe764101650 con 0x7fe74c068f60 2023-12-16T20:50:22.747 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.746+0000 7fe76a3cf700 1 -- 172.21.15.52:0/1147396965 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe74c068f60 msgr2=0x7fe74c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:22.747 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.746+0000 7fe76a3cf700 1 --2- 172.21.15.52:0/1147396965 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe74c068f60 0x7fe74c06b410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe7540135d0 tx=0x7fe754009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:22.747 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.746+0000 7fe76a3cf700 1 -- 172.21.15.52:0/1147396965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7640fcd00 msgr2=0x7fe764112760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:22.747 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.746+0000 7fe76a3cf700 1 --2- 172.21.15.52:0/1147396965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7640fcd00 0x7fe764112760 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe7580096d0 tx=0x7fe758009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:22.747 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.746+0000 7fe76a3cf700 1 -- 172.21.15.52:0/1147396965 shutdown_connections 2023-12-16T20:50:22.747 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.747+0000 7fe76a3cf700 1 --2- 172.21.15.52:0/1147396965 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fe74c068f60 0x7fe74c06b410 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:22.747 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.747+0000 7fe76a3cf700 1 --2- 172.21.15.52:0/1147396965 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7640fbaf0 0x7fe764110210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:22.748 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.747+0000 7fe76a3cf700 1 --2- 172.21.15.52:0/1147396965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7640fcd00 0x7fe764112760 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:22.748 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.747+0000 7fe76a3cf700 1 -- 172.21.15.52:0/1147396965 >> 172.21.15.52:0/1147396965 conn(0x7fe7640f7190 msgr2=0x7fe7640fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:22.748 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.747+0000 7fe76a3cf700 1 -- 172.21.15.52:0/1147396965 shutdown_connections 2023-12-16T20:50:22.748 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.747+0000 7fe76a3cf700 1 -- 172.21.15.52:0/1147396965 wait complete. 2023-12-16T20:50:22.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.854+0000 7f7ee1bd2700 1 -- 172.21.15.52:0/2432544856 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7edc0fd730 msgr2=0x7f7edc0ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:22.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.854+0000 7f7ee1bd2700 1 --2- 172.21.15.52:0/2432544856 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7edc0fd730 0x7f7edc0ffb10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc004940 tx=0x7f7ecc009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:22.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.854+0000 7f7ee1bd2700 1 -- 172.21.15.52:0/2432544856 shutdown_connections 2023-12-16T20:50:22.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.854+0000 7f7ee1bd2700 1 --2- 172.21.15.52:0/2432544856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7edc100050 0x7f7edc102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:22.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.854+0000 7f7ee1bd2700 1 --2- 172.21.15.52:0/2432544856 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7edc0fd730 0x7f7edc0ffb10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:22.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.854+0000 7f7ee1bd2700 1 -- 172.21.15.52:0/2432544856 >> 172.21.15.52:0/2432544856 conn(0x7f7edc0f71d0 msgr2=0x7f7edc0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:22.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.854+0000 7f7ee1bd2700 1 -- 172.21.15.52:0/2432544856 shutdown_connections 2023-12-16T20:50:22.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.854+0000 7f7ee1bd2700 1 -- 172.21.15.52:0/2432544856 wait complete. 2023-12-16T20:50:22.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.855+0000 7f7ee1bd2700 1 Processor -- start 2023-12-16T20:50:22.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.855+0000 7f7ee1bd2700 1 -- start start 2023-12-16T20:50:22.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.855+0000 7f7ee1bd2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7edc0fd730 0x7f7edc0fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:22.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.855+0000 7f7ee1bd2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7edc100050 0x7f7edc0fb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:22.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.855+0000 7f7ee1bd2700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc0fbc40 con 0x7f7edc100050 2023-12-16T20:50:22.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.855+0000 7f7ee1bd2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc0fbdb0 con 0x7f7edc0fd730 2023-12-16T20:50:22.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.856+0000 7f7edaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7edc100050 0x7f7edc0fb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:22.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.856+0000 7f7edaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7edc100050 0x7f7edc0fb640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37708/0 (socket says 172.21.15.52:37708) 2023-12-16T20:50:22.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.856+0000 7f7edaffd700 1 -- 172.21.15.52:0/2920775685 learned_addr learned my addr 172.21.15.52:0/2920775685 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:50:22.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.856+0000 7f7edb7fe700 1 --2- 172.21.15.52:0/2920775685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7edc0fd730 0x7f7edc0fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:22.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.856+0000 7f7edaffd700 1 -- 172.21.15.52:0/2920775685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7edc0fd730 msgr2=0x7f7edc0fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:22.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.856+0000 7f7edaffd700 1 --2- 172.21.15.52:0/2920775685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7edc0fd730 0x7f7edc0fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:22.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.856+0000 7f7edaffd700 1 -- 172.21.15.52:0/2920775685 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ecc005040 con 0x7f7edc100050 2023-12-16T20:50:22.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.856+0000 7f7edaffd700 1 --2- 172.21.15.52:0/2920775685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7edc100050 0x7f7edc0fb640 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0002a80 tx=0x7f7ed00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:22.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.856+0000 7f7ec3fff700 1 -- 172.21.15.52:0/2920775685 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ed0007cc0 con 0x7f7edc100050 2023-12-16T20:50:22.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.856+0000 7f7ee1bd2700 1 -- 172.21.15.52:0/2920775685 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7edc0fc090 con 0x7f7edc100050 2023-12-16T20:50:22.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.856+0000 7f7ee1bd2700 1 -- 172.21.15.52:0/2920775685 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7edc10d060 con 0x7f7edc100050 2023-12-16T20:50:22.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.857+0000 7f7ec3fff700 1 -- 172.21.15.52:0/2920775685 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ed000f040 con 0x7f7edc100050 2023-12-16T20:50:22.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.857+0000 7f7ec3fff700 1 -- 172.21.15.52:0/2920775685 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ed000da60 con 0x7f7edc100050 2023-12-16T20:50:22.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.858+0000 7f7ee1bd2700 1 -- 172.21.15.52:0/2920775685 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7edc04bb80 con 0x7f7edc100050 2023-12-16T20:50:22.860 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.860+0000 7f7ec3fff700 1 -- 172.21.15.52:0/2920775685 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f7ed000dbc0 con 0x7f7edc100050 2023-12-16T20:50:22.861 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.860+0000 7f7ec3fff700 1 --2- 172.21.15.52:0/2920775685 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7ec4069030 0x7f7ec406b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:22.861 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.860+0000 7f7ec3fff700 1 -- 172.21.15.52:0/2920775685 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7ed0086fb0 con 0x7f7edc100050 2023-12-16T20:50:22.861 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.861+0000 7f7edb7fe700 1 --2- 172.21.15.52:0/2920775685 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7ec4069030 0x7f7ec406b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:22.861 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.861+0000 7f7edb7fe700 1 --2- 172.21.15.52:0/2920775685 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7ec4069030 0x7f7ec406b4e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc004890 tx=0x7f7ecc009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:22.867 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:22.867+0000 7f7ec3fff700 1 -- 172.21.15.52:0/2920775685 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7ed0056ea0 con 0x7f7edc100050 2023-12-16T20:50:23.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:22 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:50:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:22 smithi052 ceph-mon[80761]: Upgrade: Target is version 19.0.0-202-gc7586e95 (unknown) 2023-12-16T20:50:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:22 smithi052 ceph-mon[80761]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5, digests ['quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5'] 2023-12-16T20:50:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:22 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:50:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:22 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:50:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:22 smithi052 ceph-mon[80761]: pgmap v153: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2023-12-16T20:50:23.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:22 smithi052 ceph-mon[80761]: Upgrade: Need to upgrade myself (mgr.smithi052.nmkcbs) 2023-12-16T20:50:23.080 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.080+0000 7f7ee1bd2700 1 -- 172.21.15.52:0/2920775685 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7edc100cb0 con 0x7f7ec4069030 2023-12-16T20:50:23.091 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.090+0000 7f7ec3fff700 1 -- 172.21.15.52:0/2920775685 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f7edc100cb0 con 0x7f7ec4069030 2023-12-16T20:50:23.091 INFO:teuthology.orchestra.run.smithi052.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T20:50:23.091 INFO:teuthology.orchestra.run.smithi052.stdout:alertmanager.smithi052 smithi052 *:9093,9094 running (3m) 45s ago 4m 25.6M - 0.25.0 c8568f914cd2 70fee28ba742 2023-12-16T20:50:23.091 INFO:teuthology.orchestra.run.smithi052.stdout:crash.smithi052 smithi052 running (4m) 45s ago 4m 6665k - 17.2.7-125-g1f2f5510 60cc83a9c8f1 2cbb0d9d4ed2 2023-12-16T20:50:23.091 INFO:teuthology.orchestra.run.smithi052.stdout:crash.smithi150 smithi150 running (3m) 49s ago 3m 6665k - 17.2.7-125-g1f2f5510 60cc83a9c8f1 bb44b459c1ab 2023-12-16T20:50:23.091 INFO:teuthology.orchestra.run.smithi052.stdout:grafana.smithi052 smithi052 *:3000 running (3m) 45s ago 4m 82.2M - 9.4.7 954c08fa6188 7a62e1cb6f33 2023-12-16T20:50:23.091 INFO:teuthology.orchestra.run.smithi052.stdout:mds.cephfs.smithi052.bvmosa smithi052 running (60s) 45s ago 60s 10.4M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 5db044ddd3dd 2023-12-16T20:50:23.091 INFO:teuthology.orchestra.run.smithi052.stdout:mds.cephfs.smithi052.szlsle smithi052 running (57s) 45s ago 57s 13.4M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 644f163c1474 2023-12-16T20:50:23.091 INFO:teuthology.orchestra.run.smithi052.stdout:mds.cephfs.smithi150.knuivd smithi150 running (59s) 49s ago 59s 16.8M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 05efa6673395 2023-12-16T20:50:23.092 INFO:teuthology.orchestra.run.smithi052.stdout:mds.cephfs.smithi150.njrgli smithi150 running (55s) 49s ago 55s 12.9M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 33a2bade442a 2023-12-16T20:50:23.092 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.smithi052.nmkcbs smithi052 *:9283 running (5m) 45s ago 5m 478M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 aedf615cc9aa 2023-12-16T20:50:23.092 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.smithi150.vjiupe smithi150 *:8443,9283 running (3m) 49s ago 3m 434M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 dcdbfd832a9b 2023-12-16T20:50:23.092 INFO:teuthology.orchestra.run.smithi052.stdout:mon.smithi052 smithi052 running (5m) 45s ago 5m 47.0M 2048M 17.2.7-125-g1f2f5510 60cc83a9c8f1 094cffecdaf4 2023-12-16T20:50:23.092 INFO:teuthology.orchestra.run.smithi052.stdout:mon.smithi150 smithi150 running (3m) 49s ago 3m 42.3M 2048M 17.2.7-125-g1f2f5510 60cc83a9c8f1 4e8a89294822 2023-12-16T20:50:23.092 INFO:teuthology.orchestra.run.smithi052.stdout:node-exporter.smithi052 smithi052 *:9100 running (4m) 45s ago 4m 17.3M - 1.5.0 0da6a335fe13 71e781c7e5ac 2023-12-16T20:50:23.092 INFO:teuthology.orchestra.run.smithi052.stdout:node-exporter.smithi150 smithi150 *:9100 running (3m) 49s ago 3m 16.9M - 1.5.0 0da6a335fe13 76e3676984fa 2023-12-16T20:50:23.092 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0 smithi052 running (3m) 45s ago 3m 48.7M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 70d69be336b3 2023-12-16T20:50:23.092 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1 smithi052 running (3m) 45s ago 3m 46.0M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 a39f1afa7295 2023-12-16T20:50:23.092 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2 smithi052 running (2m) 45s ago 2m 46.9M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 a382edce4272 2023-12-16T20:50:23.092 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3 smithi150 running (2m) 49s ago 2m 46.8M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 634f54a96481 2023-12-16T20:50:23.092 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4 smithi150 running (2m) 49s ago 2m 42.8M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 60f7c481a05d 2023-12-16T20:50:23.092 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5 smithi150 running (117s) 49s ago 116s 42.1M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 12c8aba080a8 2023-12-16T20:50:23.092 INFO:teuthology.orchestra.run.smithi052.stdout:prometheus.smithi052 smithi052 *:9095 running (3m) 45s ago 4m 43.8M - 2.43.0 a07b618ecd1d eebcaf89c62f 2023-12-16T20:50:23.094 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.093+0000 7f7ee1bd2700 1 -- 172.21.15.52:0/2920775685 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7ec4069030 msgr2=0x7f7ec406b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:23.094 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.093+0000 7f7ee1bd2700 1 --2- 172.21.15.52:0/2920775685 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7ec4069030 0x7f7ec406b4e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc004890 tx=0x7f7ecc009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:23.094 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.093+0000 7f7ee1bd2700 1 -- 172.21.15.52:0/2920775685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7edc100050 msgr2=0x7f7edc0fb640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:23.094 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.093+0000 7f7ee1bd2700 1 --2- 172.21.15.52:0/2920775685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7edc100050 0x7f7edc0fb640 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0002a80 tx=0x7f7ed00046b0 comp rx=0 tx=0).stop 2023-12-16T20:50:23.094 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.093+0000 7f7ee1bd2700 1 -- 172.21.15.52:0/2920775685 shutdown_connections 2023-12-16T20:50:23.094 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.093+0000 7f7ee1bd2700 1 --2- 172.21.15.52:0/2920775685 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f7ec4069030 0x7f7ec406b4e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:23.094 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.093+0000 7f7ee1bd2700 1 --2- 172.21.15.52:0/2920775685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7edc0fd730 0x7f7edc0fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:23.094 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.093+0000 7f7ee1bd2700 1 --2- 172.21.15.52:0/2920775685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7edc100050 0x7f7edc0fb640 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:23.094 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.093+0000 7f7ee1bd2700 1 -- 172.21.15.52:0/2920775685 >> 172.21.15.52:0/2920775685 conn(0x7f7edc0f71d0 msgr2=0x7f7edc0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:23.094 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.093+0000 7f7ee1bd2700 1 -- 172.21.15.52:0/2920775685 shutdown_connections 2023-12-16T20:50:23.095 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.093+0000 7f7ee1bd2700 1 -- 172.21.15.52:0/2920775685 wait complete. 2023-12-16T20:50:23.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:22 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:50:23.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:22 smithi150 ceph-mon[88000]: Upgrade: Target is version 19.0.0-202-gc7586e95 (unknown) 2023-12-16T20:50:23.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:22 smithi150 ceph-mon[88000]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5, digests ['quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5'] 2023-12-16T20:50:23.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:22 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:50:23.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:22 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:50:23.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:22 smithi150 ceph-mon[88000]: pgmap v153: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2023-12-16T20:50:23.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:22 smithi150 ceph-mon[88000]: Upgrade: Need to upgrade myself (mgr.smithi052.nmkcbs) 2023-12-16T20:50:23.204 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.203+0000 7f4f191cf700 1 -- 172.21.15.52:0/1078751042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f140704f0 msgr2=0x7f4f14070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:23.204 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.203+0000 7f4f191cf700 1 --2- 172.21.15.52:0/1078751042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f140704f0 0x7f4f14070960 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4f08004940 tx=0x7f4f08009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:23.205 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.203+0000 7f4f191cf700 1 -- 172.21.15.52:0/1078751042 shutdown_connections 2023-12-16T20:50:23.205 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.203+0000 7f4f191cf700 1 --2- 172.21.15.52:0/1078751042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f140704f0 0x7f4f14070960 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:23.205 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.203+0000 7f4f191cf700 1 --2- 172.21.15.52:0/1078751042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f14071b40 0x7f4f1406ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:23.205 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.203+0000 7f4f191cf700 1 -- 172.21.15.52:0/1078751042 >> 172.21.15.52:0/1078751042 conn(0x7f4f140f82d0 msgr2=0x7f4f140fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:23.205 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.204+0000 7f4f191cf700 1 -- 172.21.15.52:0/1078751042 shutdown_connections 2023-12-16T20:50:23.205 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.204+0000 7f4f191cf700 1 -- 172.21.15.52:0/1078751042 wait complete. 2023-12-16T20:50:23.206 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.205+0000 7f4f191cf700 1 Processor -- start 2023-12-16T20:50:23.206 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.205+0000 7f4f191cf700 1 -- start start 2023-12-16T20:50:23.206 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.205+0000 7f4f191cf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f140704f0 0x7f4f1410f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:23.206 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.205+0000 7f4f191cf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f14071b40 0x7f4f1410fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:23.206 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.205+0000 7f4f191cf700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f14110370 con 0x7f4f14071b40 2023-12-16T20:50:23.206 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.205+0000 7f4f191cf700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f14110c10 con 0x7f4f140704f0 2023-12-16T20:50:23.207 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.205+0000 7f4f1259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f14071b40 0x7f4f1410fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:23.207 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.205+0000 7f4f1259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f14071b40 0x7f4f1410fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37734/0 (socket says 172.21.15.52:37734) 2023-12-16T20:50:23.207 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.205+0000 7f4f1259c700 1 -- 172.21.15.52:0/3990432481 learned_addr learned my addr 172.21.15.52:0/3990432481 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:50:23.207 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.205+0000 7f4f12d9d700 1 --2- 172.21.15.52:0/3990432481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f140704f0 0x7f4f1410f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:23.207 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.206+0000 7f4f1259c700 1 -- 172.21.15.52:0/3990432481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f140704f0 msgr2=0x7f4f1410f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:23.207 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.206+0000 7f4f1259c700 1 --2- 172.21.15.52:0/3990432481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f140704f0 0x7f4f1410f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:23.207 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.206+0000 7f4f1259c700 1 -- 172.21.15.52:0/3990432481 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f08005040 con 0x7f4f14071b40 2023-12-16T20:50:23.207 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.206+0000 7f4f1259c700 1 --2- 172.21.15.52:0/3990432481 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f14071b40 0x7f4f1410fd70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4f08008f80 tx=0x7f4f08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:23.208 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.206+0000 7f4efb7fe700 1 -- 172.21.15.52:0/3990432481 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f0800e030 con 0x7f4f14071b40 2023-12-16T20:50:23.208 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.206+0000 7f4efb7fe700 1 -- 172.21.15.52:0/3990432481 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f08004290 con 0x7f4f14071b40 2023-12-16T20:50:23.208 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.206+0000 7f4efb7fe700 1 -- 172.21.15.52:0/3990432481 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f0800b410 con 0x7f4f14071b40 2023-12-16T20:50:23.208 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.206+0000 7f4f191cf700 1 -- 172.21.15.52:0/3990432481 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f14110e90 con 0x7f4f14071b40 2023-12-16T20:50:23.208 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.207+0000 7f4f191cf700 1 -- 172.21.15.52:0/3990432481 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f141113d0 con 0x7f4f14071b40 2023-12-16T20:50:23.209 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.208+0000 7f4f191cf700 1 -- 172.21.15.52:0/3990432481 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f141096e0 con 0x7f4f14071b40 2023-12-16T20:50:23.209 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.208+0000 7f4efb7fe700 1 -- 172.21.15.52:0/3990432481 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f4f0801c400 con 0x7f4f14071b40 2023-12-16T20:50:23.210 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.209+0000 7f4efb7fe700 1 --2- 172.21.15.52:0/3990432481 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4efc068f70 0x7f4efc06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:23.210 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.209+0000 7f4efb7fe700 1 -- 172.21.15.52:0/3990432481 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4f08088870 con 0x7f4f14071b40 2023-12-16T20:50:23.210 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.209+0000 7f4f12d9d700 1 --2- 172.21.15.52:0/3990432481 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4efc068f70 0x7f4efc06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:23.211 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.210+0000 7f4f12d9d700 1 --2- 172.21.15.52:0/3990432481 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4efc068f70 0x7f4efc06b420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4f04004600 tx=0x7f4f0400b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:23.215 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.214+0000 7f4efb7fe700 1 -- 172.21.15.52:0/3990432481 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4f08058820 con 0x7f4f14071b40 2023-12-16T20:50:23.496 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.495+0000 7f4f191cf700 1 -- 172.21.15.52:0/3990432481 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4f141078f0 con 0x7f4f14071b40 2023-12-16T20:50:23.496 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.496+0000 7f4efb7fe700 1 -- 172.21.15.52:0/3990432481 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f4f080583b0 con 0x7f4f14071b40 2023-12-16T20:50:23.497 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2023-12-16T20:50:23.497 INFO:teuthology.orchestra.run.smithi052.stdout: "mon": { 2023-12-16T20:50:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 2 2023-12-16T20:50:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2023-12-16T20:50:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr": { 2023-12-16T20:50:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 2 2023-12-16T20:50:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2023-12-16T20:50:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": { 2023-12-16T20:50:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 6 2023-12-16T20:50:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2023-12-16T20:50:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: "mds": { 2023-12-16T20:50:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 4 2023-12-16T20:50:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2023-12-16T20:50:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: "overall": { 2023-12-16T20:50:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 14 2023-12-16T20:50:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: } 2023-12-16T20:50:23.498 INFO:teuthology.orchestra.run.smithi052.stdout:} 2023-12-16T20:50:23.500 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.499+0000 7f4f191cf700 1 -- 172.21.15.52:0/3990432481 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4efc068f70 msgr2=0x7f4efc06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:23.500 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.499+0000 7f4f191cf700 1 --2- 172.21.15.52:0/3990432481 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4efc068f70 0x7f4efc06b420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4f04004600 tx=0x7f4f0400b040 comp rx=0 tx=0).stop 2023-12-16T20:50:23.500 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.499+0000 7f4f191cf700 1 -- 172.21.15.52:0/3990432481 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f14071b40 msgr2=0x7f4f1410fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:23.500 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.499+0000 7f4f191cf700 1 --2- 172.21.15.52:0/3990432481 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f14071b40 0x7f4f1410fd70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4f08008f80 tx=0x7f4f08009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:23.500 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.499+0000 7f4f191cf700 1 -- 172.21.15.52:0/3990432481 shutdown_connections 2023-12-16T20:50:23.501 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.499+0000 7f4f191cf700 1 --2- 172.21.15.52:0/3990432481 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f4efc068f70 0x7f4efc06b420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:23.501 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.499+0000 7f4f191cf700 1 --2- 172.21.15.52:0/3990432481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f140704f0 0x7f4f1410f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:23.501 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.499+0000 7f4f191cf700 1 --2- 172.21.15.52:0/3990432481 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f14071b40 0x7f4f1410fd70 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:23.501 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.499+0000 7f4f191cf700 1 -- 172.21.15.52:0/3990432481 >> 172.21.15.52:0/3990432481 conn(0x7f4f140f82d0 msgr2=0x7f4f14103390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:23.501 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.500+0000 7f4f191cf700 1 -- 172.21.15.52:0/3990432481 shutdown_connections 2023-12-16T20:50:23.501 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.501+0000 7f4f191cf700 1 -- 172.21.15.52:0/3990432481 wait complete. 2023-12-16T20:50:23.615 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.614+0000 7f615a3be700 1 -- 172.21.15.52:0/2952964584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61540fe230 msgr2=0x7f61540fe640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:23.615 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.614+0000 7f615a3be700 1 --2- 172.21.15.52:0/2952964584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61540fe230 0x7f61540fe640 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6144004940 tx=0x7f6144009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:23.615 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.614+0000 7f615a3be700 1 -- 172.21.15.52:0/2952964584 shutdown_connections 2023-12-16T20:50:23.615 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.614+0000 7f615a3be700 1 --2- 172.21.15.52:0/2952964584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61540fec10 0x7f61540facc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:23.615 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.614+0000 7f615a3be700 1 --2- 172.21.15.52:0/2952964584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61540fe230 0x7f61540fe640 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:23.615 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.614+0000 7f615a3be700 1 -- 172.21.15.52:0/2952964584 >> 172.21.15.52:0/2952964584 conn(0x7f61540f6950 msgr2=0x7f61540f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:23.615 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.614+0000 7f615a3be700 1 -- 172.21.15.52:0/2952964584 shutdown_connections 2023-12-16T20:50:23.615 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.614+0000 7f615a3be700 1 -- 172.21.15.52:0/2952964584 wait complete. 2023-12-16T20:50:23.616 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.615+0000 7f615a3be700 1 Processor -- start 2023-12-16T20:50:23.616 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.615+0000 7f615a3be700 1 -- start start 2023-12-16T20:50:23.616 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.615+0000 7f615a3be700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61540fe230 0x7f61540fda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:23.616 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.615+0000 7f615a3be700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61540fbef0 0x7f61540fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:23.616 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.615+0000 7f615a3be700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61540fc8f0 con 0x7f61540fe230 2023-12-16T20:50:23.616 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.615+0000 7f615a3be700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61540fca30 con 0x7f61540fbef0 2023-12-16T20:50:23.617 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.616+0000 7f6153fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61540fe230 0x7f61540fda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:23.617 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.616+0000 7f6153fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61540fe230 0x7f61540fda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37754/0 (socket says 172.21.15.52:37754) 2023-12-16T20:50:23.617 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.616+0000 7f6153fff700 1 -- 172.21.15.52:0/4035121099 learned_addr learned my addr 172.21.15.52:0/4035121099 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:50:23.617 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.616+0000 7f61537fe700 1 --2- 172.21.15.52:0/4035121099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61540fbef0 0x7f61540fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:23.617 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.616+0000 7f6153fff700 1 -- 172.21.15.52:0/4035121099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61540fbef0 msgr2=0x7f61540fc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:23.617 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.616+0000 7f6153fff700 1 --2- 172.21.15.52:0/4035121099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61540fbef0 0x7f61540fc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:23.617 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.616+0000 7f6153fff700 1 -- 172.21.15.52:0/4035121099 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6144005040 con 0x7f61540fe230 2023-12-16T20:50:23.618 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.616+0000 7f61537fe700 1 --2- 172.21.15.52:0/4035121099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61540fbef0 0x7f61540fc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T20:50:23.618 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.616+0000 7f6153fff700 1 --2- 172.21.15.52:0/4035121099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61540fe230 0x7f61540fda70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6144000c00 tx=0x7f6144009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:23.618 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.617+0000 7f6150ff9700 1 -- 172.21.15.52:0/4035121099 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6144021070 con 0x7f61540fe230 2023-12-16T20:50:23.618 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.617+0000 7f6150ff9700 1 -- 172.21.15.52:0/4035121099 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f614401c400 con 0x7f61540fe230 2023-12-16T20:50:23.628 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.617+0000 7f6150ff9700 1 -- 172.21.15.52:0/4035121099 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f614400e070 con 0x7f61540fe230 2023-12-16T20:50:23.628 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.617+0000 7f615a3be700 1 -- 172.21.15.52:0/4035121099 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61540fccb0 con 0x7f61540fe230 2023-12-16T20:50:23.628 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.617+0000 7f615a3be700 1 -- 172.21.15.52:0/4035121099 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61541158b0 con 0x7f61540fe230 2023-12-16T20:50:23.628 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.618+0000 7f615a3be700 1 -- 172.21.15.52:0/4035121099 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615404bb80 con 0x7f61540fe230 2023-12-16T20:50:23.628 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.619+0000 7f6150ff9700 1 -- 172.21.15.52:0/4035121099 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f614401c9c0 con 0x7f61540fe230 2023-12-16T20:50:23.628 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.620+0000 7f6150ff9700 1 --2- 172.21.15.52:0/4035121099 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f613c068f60 0x7f613c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:23.629 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.620+0000 7f6150ff9700 1 -- 172.21.15.52:0/4035121099 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6144088590 con 0x7f61540fe230 2023-12-16T20:50:23.629 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.620+0000 7f61537fe700 1 --2- 172.21.15.52:0/4035121099 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f613c068f60 0x7f613c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:23.629 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.620+0000 7f61537fe700 1 --2- 172.21.15.52:0/4035121099 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f613c068f60 0x7f613c06b410 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6148009b80 tx=0x7f614800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:23.629 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:23.628+0000 7f6150ff9700 1 -- 172.21.15.52:0/4035121099 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f61440585f0 con 0x7f61540fe230 2023-12-16T20:50:24.052 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:24.050+0000 7f615a3be700 1 -- 172.21.15.52:0/4035121099 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f61540fee40 con 0x7f613c068f60 2023-12-16T20:50:24.053 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:24.053+0000 7f6150ff9700 1 -- 172.21.15.52:0/4035121099 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f61540fee40 con 0x7f613c068f60 2023-12-16T20:50:24.054 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2023-12-16T20:50:24.054 INFO:teuthology.orchestra.run.smithi052.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5", 2023-12-16T20:50:24.054 INFO:teuthology.orchestra.run.smithi052.stdout: "in_progress": true, 2023-12-16T20:50:24.054 INFO:teuthology.orchestra.run.smithi052.stdout: "which": "Upgrading daemons of type(s) mgr", 2023-12-16T20:50:24.054 INFO:teuthology.orchestra.run.smithi052.stdout: "services_complete": [], 2023-12-16T20:50:24.054 INFO:teuthology.orchestra.run.smithi052.stdout: "progress": "0/2 daemons upgraded", 2023-12-16T20:50:24.054 INFO:teuthology.orchestra.run.smithi052.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5 image on host smithi150", 2023-12-16T20:50:24.054 INFO:teuthology.orchestra.run.smithi052.stdout: "is_paused": false 2023-12-16T20:50:24.054 INFO:teuthology.orchestra.run.smithi052.stdout:} 2023-12-16T20:50:24.057 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:24.056+0000 7f615a3be700 1 -- 172.21.15.52:0/4035121099 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f613c068f60 msgr2=0x7f613c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:24.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:24.056+0000 7f615a3be700 1 --2- 172.21.15.52:0/4035121099 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f613c068f60 0x7f613c06b410 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6148009b80 tx=0x7f614800b040 comp rx=0 tx=0).stop 2023-12-16T20:50:24.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:24.056+0000 7f615a3be700 1 -- 172.21.15.52:0/4035121099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61540fe230 msgr2=0x7f61540fda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:24.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:24.056+0000 7f615a3be700 1 --2- 172.21.15.52:0/4035121099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61540fe230 0x7f61540fda70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6144000c00 tx=0x7f6144009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:24.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:24.056+0000 7f615a3be700 1 -- 172.21.15.52:0/4035121099 shutdown_connections 2023-12-16T20:50:24.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:24.056+0000 7f615a3be700 1 --2- 172.21.15.52:0/4035121099 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f613c068f60 0x7f613c06b410 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:24.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:24.056+0000 7f615a3be700 1 --2- 172.21.15.52:0/4035121099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61540fe230 0x7f61540fda70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:24.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:24.056+0000 7f615a3be700 1 --2- 172.21.15.52:0/4035121099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61540fbef0 0x7f61540fc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:24.059 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:24.056+0000 7f615a3be700 1 -- 172.21.15.52:0/4035121099 >> 172.21.15.52:0/4035121099 conn(0x7f61540f6950 msgr2=0x7f6154101210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:24.059 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:24.057+0000 7f615a3be700 1 -- 172.21.15.52:0/4035121099 shutdown_connections 2023-12-16T20:50:24.059 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:24.057+0000 7f615a3be700 1 -- 172.21.15.52:0/4035121099 wait complete. 2023-12-16T20:50:24.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:23 smithi150 ceph-mon[88000]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:50:24.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:23 smithi150 ceph-mon[88000]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5 on smithi150 2023-12-16T20:50:24.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:23 smithi150 ceph-mon[88000]: from='client.24327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:50:24.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:23 smithi150 ceph-mon[88000]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:50:24.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:23 smithi150 ceph-mon[88000]: from='client.? 172.21.15.52:0/3990432481' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T20:50:24.189 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:23 smithi052 ceph-mon[80761]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:50:24.189 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:23 smithi052 ceph-mon[80761]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5 on smithi150 2023-12-16T20:50:24.189 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:23 smithi052 ceph-mon[80761]: from='client.24327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:50:24.189 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:23 smithi052 ceph-mon[80761]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:50:24.190 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:23 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/3990432481' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T20:50:25.187 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:24 smithi150 ceph-mon[88000]: pgmap v154: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 5 op/s 2023-12-16T20:50:25.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:24 smithi150 ceph-mon[88000]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:50:25.188 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:24 smithi052 ceph-mon[80761]: pgmap v154: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 5 op/s 2023-12-16T20:50:25.188 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:24 smithi052 ceph-mon[80761]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:50:27.187 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:26 smithi150 ceph-mon[88000]: pgmap v155: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2023-12-16T20:50:27.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:26 smithi052 ceph-mon[80761]: pgmap v155: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2023-12-16T20:50:28.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:28 smithi052 ceph-mon[80761]: pgmap v156: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2023-12-16T20:50:28.938 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:28 smithi150 ceph-mon[88000]: pgmap v156: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2023-12-16T20:50:31.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:30 smithi052 ceph-mon[80761]: pgmap v157: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2023-12-16T20:50:31.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:30 smithi150 ceph-mon[88000]: pgmap v157: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2023-12-16T20:50:33.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:32 smithi052 ceph-mon[80761]: pgmap v158: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2023-12-16T20:50:33.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:32 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:50:33.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:32 smithi150 ceph-mon[88000]: pgmap v158: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2023-12-16T20:50:33.438 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:32 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:50:33.786 INFO:tasks.workunit.client.1.smithi150.stderr:Updating files: 95% (11107/11620) Updating files: 96% (11156/11620) Updating files: 97% (11272/11620) Updating files: 98% (11388/11620) Updating files: 99% (11504/11620) Updating files: 100% (11620/11620) Updating files: 100% (11620/11620), done. 2023-12-16T20:50:34.795 INFO:tasks.workunit.client.1.smithi150.stderr:Note: switching to 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'. 2023-12-16T20:50:34.795 INFO:tasks.workunit.client.1.smithi150.stderr: 2023-12-16T20:50:34.795 INFO:tasks.workunit.client.1.smithi150.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-16T20:50:34.795 INFO:tasks.workunit.client.1.smithi150.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-16T20:50:34.795 INFO:tasks.workunit.client.1.smithi150.stderr:state without impacting any branches by switching back to a branch. 2023-12-16T20:50:34.795 INFO:tasks.workunit.client.1.smithi150.stderr: 2023-12-16T20:50:34.795 INFO:tasks.workunit.client.1.smithi150.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-16T20:50:34.795 INFO:tasks.workunit.client.1.smithi150.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-16T20:50:34.796 INFO:tasks.workunit.client.1.smithi150.stderr: 2023-12-16T20:50:34.796 INFO:tasks.workunit.client.1.smithi150.stderr: git switch -c 2023-12-16T20:50:34.796 INFO:tasks.workunit.client.1.smithi150.stderr: 2023-12-16T20:50:34.796 INFO:tasks.workunit.client.1.smithi150.stderr:Or undo this operation with: 2023-12-16T20:50:34.796 INFO:tasks.workunit.client.1.smithi150.stderr: 2023-12-16T20:50:34.796 INFO:tasks.workunit.client.1.smithi150.stderr: git switch - 2023-12-16T20:50:34.796 INFO:tasks.workunit.client.1.smithi150.stderr: 2023-12-16T20:50:34.796 INFO:tasks.workunit.client.1.smithi150.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-16T20:50:34.796 INFO:tasks.workunit.client.1.smithi150.stderr: 2023-12-16T20:50:34.796 INFO:tasks.workunit.client.1.smithi150.stderr:HEAD is now at c7586e95519 Merge branch 'wip-qa-container-distros-s' of https://github.com/cbodley/ceph into wip-yuri8-testing-2023-12-15-0911 2023-12-16T20:50:34.800 DEBUG:teuthology.orchestra.run.smithi150:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2023-12-16T20:50:34.859 INFO:tasks.workunit.client.1.smithi150.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-16T20:50:34.861 INFO:tasks.workunit.client.1.smithi150.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2023-12-16T20:50:34.861 INFO:tasks.workunit.client.1.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-16T20:50:35.242 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:34 smithi150 ceph-mon[88000]: pgmap v159: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2023-12-16T20:50:35.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:34 smithi052 ceph-mon[80761]: pgmap v159: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2023-12-16T20:50:35.617 INFO:tasks.workunit.client.1.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-16T20:50:35.685 INFO:tasks.workunit.client.1.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-16T20:50:35.733 INFO:tasks.workunit.client.1.smithi150.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2023-12-16T20:50:35.735 INFO:tasks.workunit.client.1.smithi150.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2023-12-16T20:50:35.735 INFO:tasks.workunit.client.1.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-16T20:50:35.776 INFO:tasks.workunit.client.1.smithi150.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2023-12-16T20:50:35.779 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-16T20:50:35.780 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2023-12-16T20:50:35.836 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2023-12-16T20:50:35.837 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2023-12-16T20:50:35.837 DEBUG:teuthology.orchestra.run.smithi150:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c7586e95519b7be76f3b6e3a4b333888baf46bc9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2023-12-16T20:50:35.905 INFO:tasks.workunit.client.1.smithi150.stderr:+ mkdir -p fsstress 2023-12-16T20:50:35.907 INFO:tasks.workunit.client.1.smithi150.stderr:+ pushd fsstress 2023-12-16T20:50:35.908 INFO:tasks.workunit.client.1.smithi150.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2023-12-16T20:50:35.908 INFO:tasks.workunit.client.1.smithi150.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2023-12-16T20:50:36.410 INFO:tasks.workunit.client.1.smithi150.stderr:+ tar xzf ltp-full.tgz 2023-12-16T20:50:37.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:36 smithi052 ceph-mon[80761]: pgmap v160: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2023-12-16T20:50:37.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:36 smithi150 ceph-mon[88000]: pgmap v160: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2023-12-16T20:50:38.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:38 smithi052 ceph-mon[80761]: pgmap v161: 65 pgs: 65 active+clean; 3.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 113 KiB/s rd, 217 KiB/s wr, 5 op/s 2023-12-16T20:50:38.937 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:38 smithi150 ceph-mon[88000]: pgmap v161: 65 pgs: 65 active+clean; 3.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 113 KiB/s rd, 217 KiB/s wr, 5 op/s 2023-12-16T20:50:41.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:40 smithi052 ceph-mon[80761]: pgmap v162: 65 pgs: 65 active+clean; 3.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 113 KiB/s rd, 217 KiB/s wr, 5 op/s 2023-12-16T20:50:41.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:40 smithi150 ceph-mon[88000]: pgmap v162: 65 pgs: 65 active+clean; 3.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 113 KiB/s rd, 217 KiB/s wr, 5 op/s 2023-12-16T20:50:43.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:42 smithi052 ceph-mon[80761]: pgmap v163: 65 pgs: 65 active+clean; 4.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 112 KiB/s rd, 316 KiB/s wr, 27 op/s 2023-12-16T20:50:43.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:42 smithi150 ceph-mon[88000]: pgmap v163: 65 pgs: 65 active+clean; 4.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 112 KiB/s rd, 316 KiB/s wr, 27 op/s 2023-12-16T20:50:45.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:44 smithi052 ceph-mon[80761]: pgmap v164: 65 pgs: 65 active+clean; 25 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 684 KiB/s rd, 2.1 MiB/s wr, 165 op/s 2023-12-16T20:50:45.437 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:44 smithi150 ceph-mon[88000]: pgmap v164: 65 pgs: 65 active+clean; 25 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 684 KiB/s rd, 2.1 MiB/s wr, 165 op/s 2023-12-16T20:50:47.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:47 smithi052 ceph-mon[80761]: pgmap v165: 65 pgs: 65 active+clean; 38 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 684 KiB/s rd, 3.3 MiB/s wr, 297 op/s 2023-12-16T20:50:47.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:47 smithi052 ceph-mon[80761]: fsmap cephfs:2 {0=cephfs.smithi052.szlsle=up:active,1=cephfs.smithi150.knuivd=up:active} 2 up:standby-replay 2023-12-16T20:50:47.331 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:47 smithi150 ceph-mon[88000]: pgmap v165: 65 pgs: 65 active+clean; 38 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 684 KiB/s rd, 3.3 MiB/s wr, 297 op/s 2023-12-16T20:50:47.331 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:47 smithi150 ceph-mon[88000]: fsmap cephfs:2 {0=cephfs.smithi052.szlsle=up:active,1=cephfs.smithi150.knuivd=up:active} 2 up:standby-replay 2023-12-16T20:50:49.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:48 smithi150 ceph-mon[88000]: Upgrade: Updating mgr.smithi150.vjiupe 2023-12-16T20:50:49.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:48 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:50:49.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:48 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vjiupe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T20:50:49.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:48 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T20:50:49.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:48 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:50:49.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:48 smithi150 ceph-mon[88000]: Deploying daemon mgr.smithi150.vjiupe on smithi150 2023-12-16T20:50:49.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:48 smithi150 ceph-mon[88000]: pgmap v166: 65 pgs: 65 active+clean; 47 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 323 op/s 2023-12-16T20:50:49.188 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:48 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:50:49.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:48 smithi052 ceph-mon[80761]: Upgrade: Updating mgr.smithi150.vjiupe 2023-12-16T20:50:49.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:48 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:50:49.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:48 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vjiupe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T20:50:49.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:48 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T20:50:49.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:48 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T20:50:49.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:48 smithi052 ceph-mon[80761]: Deploying daemon mgr.smithi150.vjiupe on smithi150 2023-12-16T20:50:49.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:48 smithi052 ceph-mon[80761]: pgmap v166: 65 pgs: 65 active+clean; 47 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 323 op/s 2023-12-16T20:50:49.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:48 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T20:50:51.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:51 smithi052 ceph-mon[80761]: pgmap v167: 65 pgs: 65 active+clean; 52 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 324 op/s 2023-12-16T20:50:51.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:51 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:50:51.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:51 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:50:51.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:51 smithi052 ceph-mon[80761]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:50:51.374 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:51 smithi150 ceph-mon[88000]: pgmap v167: 65 pgs: 65 active+clean; 52 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 324 op/s 2023-12-16T20:50:51.374 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:51 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:50:51.374 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:51 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' 2023-12-16T20:50:51.374 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:51 smithi150 ceph-mon[88000]: from='mgr.14214 172.21.15.52:0/1188208233' entity='mgr.smithi052.nmkcbs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T20:50:53.148 INFO:tasks.workunit.client.0.smithi052.stderr:Updating files: 99% (11562/11620) Updating files: 100% (11620/11620) Updating files: 100% (11620/11620), done. 2023-12-16T20:50:53.187 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:53 smithi150 ceph-mon[88000]: pgmap v168: 65 pgs: 65 active+clean; 56 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 323 op/s 2023-12-16T20:50:53.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:53 smithi052 ceph-mon[80761]: pgmap v168: 65 pgs: 65 active+clean; 56 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 323 op/s 2023-12-16T20:50:53.797 INFO:tasks.workunit.client.0.smithi052.stderr:Note: switching to 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'. 2023-12-16T20:50:53.797 INFO:tasks.workunit.client.0.smithi052.stderr: 2023-12-16T20:50:53.797 INFO:tasks.workunit.client.0.smithi052.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-16T20:50:53.797 INFO:tasks.workunit.client.0.smithi052.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-16T20:50:53.797 INFO:tasks.workunit.client.0.smithi052.stderr:state without impacting any branches by switching back to a branch. 2023-12-16T20:50:53.797 INFO:tasks.workunit.client.0.smithi052.stderr: 2023-12-16T20:50:53.797 INFO:tasks.workunit.client.0.smithi052.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-16T20:50:53.797 INFO:tasks.workunit.client.0.smithi052.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-16T20:50:53.797 INFO:tasks.workunit.client.0.smithi052.stderr: 2023-12-16T20:50:53.797 INFO:tasks.workunit.client.0.smithi052.stderr: git switch -c 2023-12-16T20:50:53.797 INFO:tasks.workunit.client.0.smithi052.stderr: 2023-12-16T20:50:53.797 INFO:tasks.workunit.client.0.smithi052.stderr:Or undo this operation with: 2023-12-16T20:50:53.798 INFO:tasks.workunit.client.0.smithi052.stderr: 2023-12-16T20:50:53.798 INFO:tasks.workunit.client.0.smithi052.stderr: git switch - 2023-12-16T20:50:53.798 INFO:tasks.workunit.client.0.smithi052.stderr: 2023-12-16T20:50:53.798 INFO:tasks.workunit.client.0.smithi052.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-16T20:50:53.798 INFO:tasks.workunit.client.0.smithi052.stderr: 2023-12-16T20:50:53.798 INFO:tasks.workunit.client.0.smithi052.stderr:HEAD is now at c7586e95519 Merge branch 'wip-qa-container-distros-s' of https://github.com/cbodley/ceph into wip-yuri8-testing-2023-12-15-0911 2023-12-16T20:50:53.802 DEBUG:teuthology.orchestra.run.smithi052:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-16T20:50:53.867 INFO:tasks.workunit.client.0.smithi052.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-16T20:50:53.869 INFO:tasks.workunit.client.0.smithi052.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-16T20:50:53.869 INFO:tasks.workunit.client.0.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-16T20:50:54.176 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.175+0000 7f900e7db700 1 -- 172.21.15.52:0/686685910 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90080fcd00 msgr2=0x7f90080fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:54.176 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.175+0000 7f900e7db700 1 --2- 172.21.15.52:0/686685910 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90080fcd00 0x7f90080fd150 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc004940 tx=0x7f8ffc009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:54.176 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.175+0000 7f900e7db700 1 -- 172.21.15.52:0/686685910 shutdown_connections 2023-12-16T20:50:54.176 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.175+0000 7f900e7db700 1 --2- 172.21.15.52:0/686685910 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90080fcd00 0x7f90080fd150 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.176 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.175+0000 7f900e7db700 1 --2- 172.21.15.52:0/686685910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90080fbaf0 0x7f90080fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.176 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.175+0000 7f900e7db700 1 -- 172.21.15.52:0/686685910 >> 172.21.15.52:0/686685910 conn(0x7f90080f71b0 msgr2=0x7f90080f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:54.176 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.175+0000 7f900e7db700 1 -- 172.21.15.52:0/686685910 shutdown_connections 2023-12-16T20:50:54.176 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.175+0000 7f900e7db700 1 -- 172.21.15.52:0/686685910 wait complete. 2023-12-16T20:50:54.176 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.175+0000 7f900e7db700 1 Processor -- start 2023-12-16T20:50:54.178 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.176+0000 7f900e7db700 1 -- start start 2023-12-16T20:50:54.178 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.176+0000 7f900e7db700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90080fbaf0 0x7f900810d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:54.178 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.176+0000 7f900e7db700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90080fcd00 0x7f900810d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:54.178 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.176+0000 7f900e7db700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900810dfe0 con 0x7f90080fcd00 2023-12-16T20:50:54.178 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.176+0000 7f900e7db700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900810e880 con 0x7f90080fbaf0 2023-12-16T20:50:54.178 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.176+0000 7f90077fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90080fcd00 0x7f900810d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:54.178 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.176+0000 7f90077fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90080fcd00 0x7f900810d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52832/0 (socket says 172.21.15.52:52832) 2023-12-16T20:50:54.178 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.176+0000 7f90077fe700 1 -- 172.21.15.52:0/1586362622 learned_addr learned my addr 172.21.15.52:0/1586362622 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:50:54.178 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.176+0000 7f9007fff700 1 --2- 172.21.15.52:0/1586362622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90080fbaf0 0x7f900810d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:54.179 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.177+0000 7f90077fe700 1 -- 172.21.15.52:0/1586362622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90080fbaf0 msgr2=0x7f900810d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:54.179 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.177+0000 7f90077fe700 1 --2- 172.21.15.52:0/1586362622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90080fbaf0 0x7f900810d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.179 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.177+0000 7f90077fe700 1 -- 172.21.15.52:0/1586362622 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc005040 con 0x7f90080fcd00 2023-12-16T20:50:54.179 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.177+0000 7f90077fe700 1 --2- 172.21.15.52:0/1586362622 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90080fcd00 0x7f900810d9e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00b3b0 tx=0x7f8ffc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:54.179 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.177+0000 7f9004ff9700 1 -- 172.21.15.52:0/1586362622 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ffc00e040 con 0x7f90080fcd00 2023-12-16T20:50:54.179 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.177+0000 7f900e7db700 1 -- 172.21.15.52:0/1586362622 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900810eb00 con 0x7f90080fcd00 2023-12-16T20:50:54.179 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.177+0000 7f900e7db700 1 -- 172.21.15.52:0/1586362622 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900810f040 con 0x7f90080fcd00 2023-12-16T20:50:54.179 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.177+0000 7f9004ff9700 1 -- 172.21.15.52:0/1586362622 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ffc00bc50 con 0x7f90080fcd00 2023-12-16T20:50:54.179 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.177+0000 7f9004ff9700 1 -- 172.21.15.52:0/1586362622 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ffc01c630 con 0x7f90080fcd00 2023-12-16T20:50:54.180 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.178+0000 7f900e7db700 1 -- 172.21.15.52:0/1586362622 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f900804bb80 con 0x7f90080fcd00 2023-12-16T20:50:54.180 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.179+0000 7f9004ff9700 1 -- 172.21.15.52:0/1586362622 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f8ffc013420 con 0x7f90080fcd00 2023-12-16T20:50:54.180 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.179+0000 7f9004ff9700 1 --2- 172.21.15.52:0/1586362622 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ff0068f70 0x7f8ff006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:54.180 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.180+0000 7f9007fff700 1 --2- 172.21.15.52:0/1586362622 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ff0068f70 0x7f8ff006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:54.180 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.180+0000 7f9004ff9700 1 -- 172.21.15.52:0/1586362622 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8ffc088c70 con 0x7f90080fcd00 2023-12-16T20:50:54.181 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.180+0000 7f9007fff700 1 --2- 172.21.15.52:0/1586362622 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ff0068f70 0x7f8ff006b420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8009fd0 tx=0x7f8ff8011040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:54.187 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.187+0000 7f9004ff9700 1 -- 172.21.15.52:0/1586362622 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8ffc058c20 con 0x7f90080fcd00 2023-12-16T20:50:54.429 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.428+0000 7f900e7db700 1 -- 172.21.15.52:0/1586362622 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9008066310 con 0x7f8ff0068f70 2023-12-16T20:50:54.430 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.430+0000 7f9004ff9700 1 -- 172.21.15.52:0/1586362622 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7f9008066310 con 0x7f8ff0068f70 2023-12-16T20:50:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.432+0000 7f900e7db700 1 -- 172.21.15.52:0/1586362622 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ff0068f70 msgr2=0x7f8ff006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.432+0000 7f900e7db700 1 --2- 172.21.15.52:0/1586362622 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ff0068f70 0x7f8ff006b420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8009fd0 tx=0x7f8ff8011040 comp rx=0 tx=0).stop 2023-12-16T20:50:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.432+0000 7f900e7db700 1 -- 172.21.15.52:0/1586362622 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90080fcd00 msgr2=0x7f900810d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.432+0000 7f900e7db700 1 --2- 172.21.15.52:0/1586362622 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90080fcd00 0x7f900810d9e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00b3b0 tx=0x7f8ffc009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.432+0000 7f900e7db700 1 -- 172.21.15.52:0/1586362622 shutdown_connections 2023-12-16T20:50:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.432+0000 7f900e7db700 1 --2- 172.21.15.52:0/1586362622 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f8ff0068f70 0x7f8ff006b420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.432+0000 7f900e7db700 1 --2- 172.21.15.52:0/1586362622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90080fbaf0 0x7f900810d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.432+0000 7f900e7db700 1 --2- 172.21.15.52:0/1586362622 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90080fcd00 0x7f900810d9e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.432+0000 7f900e7db700 1 -- 172.21.15.52:0/1586362622 >> 172.21.15.52:0/1586362622 conn(0x7f90080f71b0 msgr2=0x7f9008065c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.432+0000 7f900e7db700 1 -- 172.21.15.52:0/1586362622 shutdown_connections 2023-12-16T20:50:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.432+0000 7f900e7db700 1 -- 172.21.15.52:0/1586362622 wait complete. 2023-12-16T20:50:54.444 INFO:teuthology.orchestra.run.smithi052.stdout:true 2023-12-16T20:50:54.454 INFO:tasks.workunit.client.1.smithi150.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2023-12-16T20:50:54.461 INFO:tasks.workunit.client.1.smithi150.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2023-12-16T20:50:54.461 INFO:tasks.workunit.client.1.smithi150.stderr:+ make 2023-12-16T20:50:54.524 INFO:tasks.workunit.client.1.smithi150.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2023-12-16T20:50:54.549 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.548+0000 7fb7a7979700 1 -- 172.21.15.52:0/3474805791 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7a00faed0 msgr2=0x7fb7a00fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:54.549 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.548+0000 7fb7a7979700 1 --2- 172.21.15.52:0/3474805791 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7a00faed0 0x7fb7a00fb2e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb79c004940 tx=0x7fb79c009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:54.549 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.548+0000 7fb7a7979700 1 -- 172.21.15.52:0/3474805791 shutdown_connections 2023-12-16T20:50:54.549 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.548+0000 7fb7a7979700 1 --2- 172.21.15.52:0/3474805791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a00fdf60 0x7fb7a00fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.549 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.548+0000 7fb7a7979700 1 --2- 172.21.15.52:0/3474805791 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7a00faed0 0x7fb7a00fb2e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.550 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.548+0000 7fb7a7979700 1 -- 172.21.15.52:0/3474805791 >> 172.21.15.52:0/3474805791 conn(0x7fb7a00f6950 msgr2=0x7fb7a00f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:54.550 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.548+0000 7fb7a7979700 1 -- 172.21.15.52:0/3474805791 shutdown_connections 2023-12-16T20:50:54.550 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.548+0000 7fb7a7979700 1 -- 172.21.15.52:0/3474805791 wait complete. 2023-12-16T20:50:54.550 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.549+0000 7fb7a7979700 1 Processor -- start 2023-12-16T20:50:54.551 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.549+0000 7fb7a7979700 1 -- start start 2023-12-16T20:50:54.551 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.549+0000 7fb7a7979700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a00faed0 0x7fb7a00fd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:54.551 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.549+0000 7fb7a7979700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7a00fdf60 0x7fb7a00fbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:54.551 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.549+0000 7fb7a7979700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a00fc470 con 0x7fb7a00fdf60 2023-12-16T20:50:54.551 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.549+0000 7fb7a7979700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a00fc5e0 con 0x7fb7a00faed0 2023-12-16T20:50:54.551 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.549+0000 7fb7a4f14700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7a00fdf60 0x7fb7a00fbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:54.551 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.549+0000 7fb7a4f14700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7a00fdf60 0x7fb7a00fbe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52854/0 (socket says 172.21.15.52:52854) 2023-12-16T20:50:54.551 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.549+0000 7fb7a4f14700 1 -- 172.21.15.52:0/4261649161 learned_addr learned my addr 172.21.15.52:0/4261649161 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:50:54.552 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.549+0000 7fb7a5715700 1 --2- 172.21.15.52:0/4261649161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a00faed0 0x7fb7a00fd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:54.552 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.550+0000 7fb7a4f14700 1 -- 172.21.15.52:0/4261649161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a00faed0 msgr2=0x7fb7a00fd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:54.552 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.550+0000 7fb7a4f14700 1 --2- 172.21.15.52:0/4261649161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a00faed0 0x7fb7a00fd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.552 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.550+0000 7fb7a4f14700 1 -- 172.21.15.52:0/4261649161 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c005040 con 0x7fb7a00fdf60 2023-12-16T20:50:54.552 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.550+0000 7fb7a4f14700 1 --2- 172.21.15.52:0/4261649161 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7a00fdf60 0x7fb7a00fbe70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb790002a50 tx=0x7fb7900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:54.552 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.550+0000 7fb795ffb700 1 -- 172.21.15.52:0/4261649161 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb790007cc0 con 0x7fb7a00fdf60 2023-12-16T20:50:54.552 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.550+0000 7fb795ffb700 1 -- 172.21.15.52:0/4261649161 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb79000f040 con 0x7fb7a00fdf60 2023-12-16T20:50:54.552 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.550+0000 7fb7a7979700 1 -- 172.21.15.52:0/4261649161 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a00fc8c0 con 0x7fb7a00fdf60 2023-12-16T20:50:54.552 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.550+0000 7fb795ffb700 1 -- 172.21.15.52:0/4261649161 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb79000d9c0 con 0x7fb7a00fdf60 2023-12-16T20:50:54.552 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.550+0000 7fb7a7979700 1 -- 172.21.15.52:0/4261649161 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a0115600 con 0x7fb7a00fdf60 2023-12-16T20:50:54.553 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.551+0000 7fb7a7979700 1 -- 172.21.15.52:0/4261649161 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a01075f0 con 0x7fb7a00fdf60 2023-12-16T20:50:54.553 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.551+0000 7fb795ffb700 1 -- 172.21.15.52:0/4261649161 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb79000db20 con 0x7fb7a00fdf60 2023-12-16T20:50:54.553 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.552+0000 7fb795ffb700 1 --2- 172.21.15.52:0/4261649161 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb78c068f60 0x7fb78c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:54.553 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.552+0000 7fb795ffb700 1 -- 172.21.15.52:0/4261649161 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb790086b20 con 0x7fb7a00fdf60 2023-12-16T20:50:54.554 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.552+0000 7fb7a5715700 1 --2- 172.21.15.52:0/4261649161 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb78c068f60 0x7fb78c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:54.554 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.552+0000 7fb7a5715700 1 --2- 172.21.15.52:0/4261649161 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb78c068f60 0x7fb78c06b410 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb79c004890 tx=0x7fb79c009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:54.557 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.556+0000 7fb795ffb700 1 -- 172.21.15.52:0/4261649161 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb7900569b0 con 0x7fb7a00fdf60 2023-12-16T20:50:54.773 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.772+0000 7fb7a7979700 1 -- 172.21.15.52:0/4261649161 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7a002b180 con 0x7fb78c068f60 2023-12-16T20:50:54.774 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.774+0000 7fb795ffb700 1 -- 172.21.15.52:0/4261649161 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7fb7a002b180 con 0x7fb78c068f60 2023-12-16T20:50:54.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.776+0000 7fb7a7979700 1 -- 172.21.15.52:0/4261649161 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb78c068f60 msgr2=0x7fb78c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:54.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.776+0000 7fb7a7979700 1 --2- 172.21.15.52:0/4261649161 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb78c068f60 0x7fb78c06b410 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb79c004890 tx=0x7fb79c009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:54.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.776+0000 7fb7a7979700 1 -- 172.21.15.52:0/4261649161 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7a00fdf60 msgr2=0x7fb7a00fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:54.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.776+0000 7fb7a7979700 1 --2- 172.21.15.52:0/4261649161 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7a00fdf60 0x7fb7a00fbe70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb790002a50 tx=0x7fb7900046b0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.776+0000 7fb7a7979700 1 -- 172.21.15.52:0/4261649161 shutdown_connections 2023-12-16T20:50:54.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.776+0000 7fb7a7979700 1 --2- 172.21.15.52:0/4261649161 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fb78c068f60 0x7fb78c06b410 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.776+0000 7fb7a7979700 1 --2- 172.21.15.52:0/4261649161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a00faed0 0x7fb7a00fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.776+0000 7fb7a7979700 1 --2- 172.21.15.52:0/4261649161 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7a00fdf60 0x7fb7a00fbe70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.776+0000 7fb7a7979700 1 -- 172.21.15.52:0/4261649161 >> 172.21.15.52:0/4261649161 conn(0x7fb7a00f6950 msgr2=0x7fb7a00f8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:54.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.776+0000 7fb7a7979700 1 -- 172.21.15.52:0/4261649161 shutdown_connections 2023-12-16T20:50:54.777 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.776+0000 7fb7a7979700 1 -- 172.21.15.52:0/4261649161 wait complete. 2023-12-16T20:50:54.881 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.880+0000 7fce59989700 1 -- 172.21.15.52:0/232126180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce540ffd70 msgr2=0x7fce541021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:54.881 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.880+0000 7fce59989700 1 --2- 172.21.15.52:0/232126180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce540ffd70 0x7fce541021d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fce48004940 tx=0x7fce48009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:54.881 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.880+0000 7fce59989700 1 -- 172.21.15.52:0/232126180 shutdown_connections 2023-12-16T20:50:54.881 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.880+0000 7fce59989700 1 --2- 172.21.15.52:0/232126180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce540ffd70 0x7fce541021d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.881 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.880+0000 7fce59989700 1 --2- 172.21.15.52:0/232126180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce54066920 0x7fce540ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.881 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.880+0000 7fce59989700 1 -- 172.21.15.52:0/232126180 >> 172.21.15.52:0/232126180 conn(0x7fce540f71d0 msgr2=0x7fce540f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:54.881 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.880+0000 7fce59989700 1 -- 172.21.15.52:0/232126180 shutdown_connections 2023-12-16T20:50:54.882 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.880+0000 7fce59989700 1 -- 172.21.15.52:0/232126180 wait complete. 2023-12-16T20:50:54.883 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.881+0000 7fce59989700 1 Processor -- start 2023-12-16T20:50:54.883 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.881+0000 7fce59989700 1 -- start start 2023-12-16T20:50:54.883 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.881+0000 7fce59989700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce54066920 0x7fce5410d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:54.883 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.881+0000 7fce59989700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce540ffd70 0x7fce5410d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:54.883 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.881+0000 7fce59989700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5410dff0 con 0x7fce54066920 2023-12-16T20:50:54.883 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.881+0000 7fce59989700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5410e890 con 0x7fce540ffd70 2023-12-16T20:50:54.883 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.881+0000 7fce52ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce54066920 0x7fce5410d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:54.883 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.881+0000 7fce52ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce54066920 0x7fce5410d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52878/0 (socket says 172.21.15.52:52878) 2023-12-16T20:50:54.884 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.881+0000 7fce52ffd700 1 -- 172.21.15.52:0/490387299 learned_addr learned my addr 172.21.15.52:0/490387299 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:50:54.884 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.881+0000 7fce527fc700 1 --2- 172.21.15.52:0/490387299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce540ffd70 0x7fce5410d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:54.884 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.882+0000 7fce52ffd700 1 -- 172.21.15.52:0/490387299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce540ffd70 msgr2=0x7fce5410d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:54.884 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.882+0000 7fce52ffd700 1 --2- 172.21.15.52:0/490387299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce540ffd70 0x7fce5410d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:54.884 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.882+0000 7fce52ffd700 1 -- 172.21.15.52:0/490387299 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce48005040 con 0x7fce54066920 2023-12-16T20:50:54.884 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.882+0000 7fce52ffd700 1 --2- 172.21.15.52:0/490387299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce54066920 0x7fce5410d4b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fce44002a80 tx=0x7fce440046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:54.884 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.882+0000 7fce3b7fe700 1 -- 172.21.15.52:0/490387299 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce44007cc0 con 0x7fce54066920 2023-12-16T20:50:54.884 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.882+0000 7fce3b7fe700 1 -- 172.21.15.52:0/490387299 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce4400f040 con 0x7fce54066920 2023-12-16T20:50:54.884 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.882+0000 7fce59989700 1 -- 172.21.15.52:0/490387299 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5410eb70 con 0x7fce54066920 2023-12-16T20:50:54.884 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.882+0000 7fce3b7fe700 1 -- 172.21.15.52:0/490387299 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce4400da60 con 0x7fce54066920 2023-12-16T20:50:54.884 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.882+0000 7fce59989700 1 -- 172.21.15.52:0/490387299 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5410f0b0 con 0x7fce54066920 2023-12-16T20:50:54.885 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.883+0000 7fce59989700 1 -- 172.21.15.52:0/490387299 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5404bb80 con 0x7fce54066920 2023-12-16T20:50:54.885 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.884+0000 7fce3b7fe700 1 -- 172.21.15.52:0/490387299 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fce4400dbc0 con 0x7fce54066920 2023-12-16T20:50:54.885 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.884+0000 7fce3b7fe700 1 --2- 172.21.15.52:0/490387299 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fce3c068f70 0x7fce3c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:54.885 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.884+0000 7fce3b7fe700 1 -- 172.21.15.52:0/490387299 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fce44086a50 con 0x7fce54066920 2023-12-16T20:50:54.885 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.884+0000 7fce527fc700 1 --2- 172.21.15.52:0/490387299 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fce3c068f70 0x7fce3c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:54.886 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.885+0000 7fce527fc700 1 --2- 172.21.15.52:0/490387299 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fce3c068f70 0x7fce3c06b420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fce4800b7d0 tx=0x7fce48009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:54.889 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:54.889+0000 7fce3b7fe700 1 -- 172.21.15.52:0/490387299 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fce440568e0 con 0x7fce54066920 2023-12-16T20:50:55.093 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.093+0000 7fce59989700 1 -- 172.21.15.52:0/490387299 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fce5410f4d0 con 0x7fce3c068f70 2023-12-16T20:50:55.103 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.102+0000 7fce3b7fe700 1 -- 172.21.15.52:0/490387299 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fce5410f4d0 con 0x7fce3c068f70 2023-12-16T20:50:55.103 INFO:teuthology.orchestra.run.smithi052.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T20:50:55.103 INFO:teuthology.orchestra.run.smithi052.stdout:alertmanager.smithi052 smithi052 *:9093,9094 running (4m) 77s ago 5m 25.6M - 0.25.0 c8568f914cd2 70fee28ba742 2023-12-16T20:50:55.103 INFO:teuthology.orchestra.run.smithi052.stdout:crash.smithi052 smithi052 running (5m) 77s ago 5m 6665k - 17.2.7-125-g1f2f5510 60cc83a9c8f1 2cbb0d9d4ed2 2023-12-16T20:50:55.103 INFO:teuthology.orchestra.run.smithi052.stdout:crash.smithi150 smithi150 running (4m) 81s ago 4m 6665k - 17.2.7-125-g1f2f5510 60cc83a9c8f1 bb44b459c1ab 2023-12-16T20:50:55.103 INFO:teuthology.orchestra.run.smithi052.stdout:grafana.smithi052 smithi052 *:3000 running (4m) 77s ago 4m 82.2M - 9.4.7 954c08fa6188 7a62e1cb6f33 2023-12-16T20:50:55.103 INFO:teuthology.orchestra.run.smithi052.stdout:mds.cephfs.smithi052.bvmosa smithi052 running (92s) 77s ago 92s 10.4M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 5db044ddd3dd 2023-12-16T20:50:55.103 INFO:teuthology.orchestra.run.smithi052.stdout:mds.cephfs.smithi052.szlsle smithi052 running (89s) 77s ago 89s 13.4M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 644f163c1474 2023-12-16T20:50:55.103 INFO:teuthology.orchestra.run.smithi052.stdout:mds.cephfs.smithi150.knuivd smithi150 running (91s) 81s ago 91s 16.8M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 05efa6673395 2023-12-16T20:50:55.104 INFO:teuthology.orchestra.run.smithi052.stdout:mds.cephfs.smithi150.njrgli smithi150 running (87s) 81s ago 87s 12.9M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 33a2bade442a 2023-12-16T20:50:55.104 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.smithi052.nmkcbs smithi052 *:9283 running (6m) 77s ago 6m 478M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 aedf615cc9aa 2023-12-16T20:50:55.104 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.smithi150.vjiupe smithi150 *:8443,9283 starting - - - - 2023-12-16T20:50:55.104 INFO:teuthology.orchestra.run.smithi052.stdout:mon.smithi052 smithi052 running (6m) 77s ago 6m 47.0M 2048M 17.2.7-125-g1f2f5510 60cc83a9c8f1 094cffecdaf4 2023-12-16T20:50:55.104 INFO:teuthology.orchestra.run.smithi052.stdout:mon.smithi150 smithi150 running (4m) 81s ago 4m 42.3M 2048M 17.2.7-125-g1f2f5510 60cc83a9c8f1 4e8a89294822 2023-12-16T20:50:55.104 INFO:teuthology.orchestra.run.smithi052.stdout:node-exporter.smithi052 smithi052 *:9100 running (5m) 77s ago 5m 17.3M - 1.5.0 0da6a335fe13 71e781c7e5ac 2023-12-16T20:50:55.104 INFO:teuthology.orchestra.run.smithi052.stdout:node-exporter.smithi150 smithi150 *:9100 running (4m) 81s ago 4m 16.9M - 1.5.0 0da6a335fe13 76e3676984fa 2023-12-16T20:50:55.104 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0 smithi052 running (3m) 77s ago 3m 48.7M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 70d69be336b3 2023-12-16T20:50:55.104 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1 smithi052 running (3m) 77s ago 3m 46.0M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 a39f1afa7295 2023-12-16T20:50:55.104 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2 smithi052 running (3m) 77s ago 3m 46.9M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 a382edce4272 2023-12-16T20:50:55.104 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3 smithi150 running (3m) 81s ago 3m 46.8M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 634f54a96481 2023-12-16T20:50:55.104 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4 smithi150 running (2m) 81s ago 2m 42.8M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 60f7c481a05d 2023-12-16T20:50:55.105 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5 smithi150 running (2m) 81s ago 2m 42.1M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 12c8aba080a8 2023-12-16T20:50:55.105 INFO:teuthology.orchestra.run.smithi052.stdout:prometheus.smithi052 smithi052 *:9095 running (3m) 77s ago 4m 43.8M - 2.43.0 a07b618ecd1d eebcaf89c62f 2023-12-16T20:50:55.106 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.105+0000 7fce59989700 1 -- 172.21.15.52:0/490387299 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fce3c068f70 msgr2=0x7fce3c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:55.106 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.105+0000 7fce59989700 1 --2- 172.21.15.52:0/490387299 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fce3c068f70 0x7fce3c06b420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fce4800b7d0 tx=0x7fce48009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:55.106 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.105+0000 7fce59989700 1 -- 172.21.15.52:0/490387299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce54066920 msgr2=0x7fce5410d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:55.106 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.105+0000 7fce59989700 1 --2- 172.21.15.52:0/490387299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce54066920 0x7fce5410d4b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fce44002a80 tx=0x7fce440046b0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.106 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.105+0000 7fce59989700 1 -- 172.21.15.52:0/490387299 shutdown_connections 2023-12-16T20:50:55.106 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.105+0000 7fce59989700 1 --2- 172.21.15.52:0/490387299 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7fce3c068f70 0x7fce3c06b420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.106 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.105+0000 7fce59989700 1 --2- 172.21.15.52:0/490387299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce54066920 0x7fce5410d4b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.106 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.105+0000 7fce59989700 1 --2- 172.21.15.52:0/490387299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce540ffd70 0x7fce5410d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.106 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.105+0000 7fce59989700 1 -- 172.21.15.52:0/490387299 >> 172.21.15.52:0/490387299 conn(0x7fce540f71d0 msgr2=0x7fce540f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:55.106 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.105+0000 7fce59989700 1 -- 172.21.15.52:0/490387299 shutdown_connections 2023-12-16T20:50:55.106 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.105+0000 7fce59989700 1 -- 172.21.15.52:0/490387299 wait complete. 2023-12-16T20:50:55.211 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.209+0000 7f2dd8126700 1 -- 172.21.15.52:0/3133307337 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dd00fbaf0 msgr2=0x7f2dd00fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:55.211 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.209+0000 7f2dd8126700 1 --2- 172.21.15.52:0/3133307337 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dd00fbaf0 0x7f2dd00fbf00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc004940 tx=0x7f2dcc009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:55.211 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.209+0000 7f2dd8126700 1 -- 172.21.15.52:0/3133307337 shutdown_connections 2023-12-16T20:50:55.211 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.209+0000 7f2dd8126700 1 --2- 172.21.15.52:0/3133307337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dd00fcd00 0x7f2dd00fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.211 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.209+0000 7f2dd8126700 1 --2- 172.21.15.52:0/3133307337 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dd00fbaf0 0x7f2dd00fbf00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.212 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.209+0000 7f2dd8126700 1 -- 172.21.15.52:0/3133307337 >> 172.21.15.52:0/3133307337 conn(0x7f2dd00f7190 msgr2=0x7f2dd00f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:55.212 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.209+0000 7f2dd8126700 1 -- 172.21.15.52:0/3133307337 shutdown_connections 2023-12-16T20:50:55.212 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.209+0000 7f2dd8126700 1 -- 172.21.15.52:0/3133307337 wait complete. 2023-12-16T20:50:55.212 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.210+0000 7f2dd8126700 1 Processor -- start 2023-12-16T20:50:55.212 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.210+0000 7f2dd8126700 1 -- start start 2023-12-16T20:50:55.212 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.210+0000 7f2dd8126700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dd00fbaf0 0x7f2dd010d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:55.212 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.210+0000 7f2dd8126700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dd00fcd00 0x7f2dd010daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:55.212 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.210+0000 7f2dd8126700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd010e0f0 con 0x7f2dd00fcd00 2023-12-16T20:50:55.212 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.210+0000 7f2dd8126700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd010e990 con 0x7f2dd00fbaf0 2023-12-16T20:50:55.212 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.210+0000 7f2dd56c1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dd00fcd00 0x7f2dd010daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:55.212 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.210+0000 7f2dd5ec2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dd00fbaf0 0x7f2dd010d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:55.212 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.210+0000 7f2dd56c1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dd00fcd00 0x7f2dd010daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52898/0 (socket says 172.21.15.52:52898) 2023-12-16T20:50:55.212 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.210+0000 7f2dd56c1700 1 -- 172.21.15.52:0/1332963085 learned_addr learned my addr 172.21.15.52:0/1332963085 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:50:55.212 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.210+0000 7f2dd56c1700 1 -- 172.21.15.52:0/1332963085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dd00fbaf0 msgr2=0x7f2dd010d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:55.212 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.210+0000 7f2dd56c1700 1 --2- 172.21.15.52:0/1332963085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dd00fbaf0 0x7f2dd010d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.210+0000 7f2dd56c1700 1 -- 172.21.15.52:0/1332963085 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc005040 con 0x7f2dd00fcd00 2023-12-16T20:50:55.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.210+0000 7f2dd56c1700 1 --2- 172.21.15.52:0/1332963085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dd00fcd00 0x7f2dd010daf0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0002a50 tx=0x7f2dc00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:55.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.211+0000 7f2dc67fc700 1 -- 172.21.15.52:0/1332963085 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dc0007cc0 con 0x7f2dd00fcd00 2023-12-16T20:50:55.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.211+0000 7f2dd8126700 1 -- 172.21.15.52:0/1332963085 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd010ec70 con 0x7f2dd00fcd00 2023-12-16T20:50:55.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.211+0000 7f2dc67fc700 1 -- 172.21.15.52:0/1332963085 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2dc000f040 con 0x7f2dd00fcd00 2023-12-16T20:50:55.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.211+0000 7f2dc67fc700 1 -- 172.21.15.52:0/1332963085 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dc000d9c0 con 0x7f2dd00fcd00 2023-12-16T20:50:55.213 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.211+0000 7f2dd8126700 1 -- 172.21.15.52:0/1332963085 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd010f1b0 con 0x7f2dd00fcd00 2023-12-16T20:50:55.214 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.212+0000 7f2dd8126700 1 -- 172.21.15.52:0/1332963085 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd01075e0 con 0x7f2dd00fcd00 2023-12-16T20:50:55.214 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.212+0000 7f2dc67fc700 1 -- 172.21.15.52:0/1332963085 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f2dc000db60 con 0x7f2dd00fcd00 2023-12-16T20:50:55.214 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.213+0000 7f2dc67fc700 1 --2- 172.21.15.52:0/1332963085 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2dbc068f60 0x7f2dbc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:55.214 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.213+0000 7f2dc67fc700 1 -- 172.21.15.52:0/1332963085 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2dc0086470 con 0x7f2dd00fcd00 2023-12-16T20:50:55.214 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.213+0000 7f2dd5ec2700 1 --2- 172.21.15.52:0/1332963085 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2dbc068f60 0x7f2dbc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:55.214 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.213+0000 7f2dd5ec2700 1 --2- 172.21.15.52:0/1332963085 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2dbc068f60 0x7f2dbc06b410 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc004890 tx=0x7f2dcc009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:55.217 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.217+0000 7f2dc67fc700 1 -- 172.21.15.52:0/1332963085 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2dc004ca00 con 0x7f2dd00fcd00 2023-12-16T20:50:55.340 INFO:tasks.workunit.client.0.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-16T20:50:55.410 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Dec 16 20:50:55 smithi150 ceph-mon[88000]: pgmap v169: 65 pgs: 65 active+clean; 63 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 306 op/s 2023-12-16T20:50:55.439 INFO:tasks.workunit.client.0.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-16T20:50:55.480 INFO:tasks.workunit.client.0.smithi052.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-16T20:50:55.481 INFO:tasks.workunit.client.0.smithi052.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-16T20:50:55.481 INFO:tasks.workunit.client.0.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-16T20:50:55.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:55 smithi052 ceph-mon[80761]: pgmap v169: 65 pgs: 65 active+clean; 63 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 306 op/s 2023-12-16T20:50:55.521 INFO:tasks.workunit.client.0.smithi052.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-16T20:50:55.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.521+0000 7f2dd8126700 1 -- 172.21.15.52:0/1332963085 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2dd004bb80 con 0x7f2dd00fcd00 2023-12-16T20:50:55.522 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.522+0000 7f2dc67fc700 1 -- 172.21.15.52:0/1332963085 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f2dc0056300 con 0x7f2dd00fcd00 2023-12-16T20:50:55.522 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2023-12-16T20:50:55.522 INFO:teuthology.orchestra.run.smithi052.stdout: "mon": { 2023-12-16T20:50:55.523 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 2 2023-12-16T20:50:55.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2023-12-16T20:50:55.523 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr": { 2023-12-16T20:50:55.523 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 2 2023-12-16T20:50:55.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2023-12-16T20:50:55.523 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": { 2023-12-16T20:50:55.523 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 6 2023-12-16T20:50:55.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2023-12-16T20:50:55.523 INFO:teuthology.orchestra.run.smithi052.stdout: "mds": { 2023-12-16T20:50:55.523 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 4 2023-12-16T20:50:55.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2023-12-16T20:50:55.523 INFO:teuthology.orchestra.run.smithi052.stdout: "overall": { 2023-12-16T20:50:55.523 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 14 2023-12-16T20:50:55.523 INFO:teuthology.orchestra.run.smithi052.stdout: } 2023-12-16T20:50:55.524 INFO:teuthology.orchestra.run.smithi052.stdout:} 2023-12-16T20:50:55.525 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T20:50:55.525 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-16T20:50:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.524+0000 7f2dd8126700 1 -- 172.21.15.52:0/1332963085 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2dbc068f60 msgr2=0x7f2dbc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.524+0000 7f2dd8126700 1 --2- 172.21.15.52:0/1332963085 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2dbc068f60 0x7f2dbc06b410 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc004890 tx=0x7f2dcc009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.524+0000 7f2dd8126700 1 -- 172.21.15.52:0/1332963085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dd00fcd00 msgr2=0x7f2dd010daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.524+0000 7f2dd8126700 1 --2- 172.21.15.52:0/1332963085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dd00fcd00 0x7f2dd010daf0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0002a50 tx=0x7f2dc00046b0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.524+0000 7f2dd8126700 1 -- 172.21.15.52:0/1332963085 shutdown_connections 2023-12-16T20:50:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.524+0000 7f2dd8126700 1 --2- 172.21.15.52:0/1332963085 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f2dbc068f60 0x7f2dbc06b410 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.524+0000 7f2dd8126700 1 --2- 172.21.15.52:0/1332963085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dd00fbaf0 0x7f2dd010d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.524+0000 7f2dd8126700 1 --2- 172.21.15.52:0/1332963085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dd00fcd00 0x7f2dd010daf0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.524+0000 7f2dd8126700 1 -- 172.21.15.52:0/1332963085 >> 172.21.15.52:0/1332963085 conn(0x7f2dd00f7190 msgr2=0x7f2dd00f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.524+0000 7f2dd8126700 1 -- 172.21.15.52:0/1332963085 shutdown_connections 2023-12-16T20:50:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.524+0000 7f2dd8126700 1 -- 172.21.15.52:0/1332963085 wait complete. 2023-12-16T20:50:55.542 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2023-12-16T20:50:55.543 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2023-12-16T20:50:55.543 DEBUG:teuthology.orchestra.run.smithi052:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c7586e95519b7be76f3b6e3a4b333888baf46bc9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2023-12-16T20:50:55.603 INFO:tasks.workunit.client.0.smithi052.stderr:+ mkdir -p fsstress 2023-12-16T20:50:55.604 INFO:tasks.workunit.client.0.smithi052.stderr:+ pushd fsstress 2023-12-16T20:50:55.605 INFO:tasks.workunit.client.0.smithi052.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2023-12-16T20:50:55.605 INFO:tasks.workunit.client.0.smithi052.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2023-12-16T20:50:55.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.631+0000 7f559d8fd700 1 -- 172.21.15.52:0/1005358160 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55980fe230 msgr2=0x7f55980fe640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:55.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.631+0000 7f559d8fd700 1 --2- 172.21.15.52:0/1005358160 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55980fe230 0x7f55980fe640 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5588004940 tx=0x7f5588009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:55.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.631+0000 7f559d8fd700 1 -- 172.21.15.52:0/1005358160 shutdown_connections 2023-12-16T20:50:55.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.631+0000 7f559d8fd700 1 --2- 172.21.15.52:0/1005358160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55980feb80 0x7f55980facc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.631+0000 7f559d8fd700 1 --2- 172.21.15.52:0/1005358160 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55980fe230 0x7f55980fe640 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.631+0000 7f559d8fd700 1 -- 172.21.15.52:0/1005358160 >> 172.21.15.52:0/1005358160 conn(0x7f55980f69d0 msgr2=0x7f55980f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:55.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.631+0000 7f559d8fd700 1 -- 172.21.15.52:0/1005358160 shutdown_connections 2023-12-16T20:50:55.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.631+0000 7f559d8fd700 1 -- 172.21.15.52:0/1005358160 wait complete. 2023-12-16T20:50:55.634 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.632+0000 7f559d8fd700 1 Processor -- start 2023-12-16T20:50:55.634 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.632+0000 7f559d8fd700 1 -- start start 2023-12-16T20:50:55.634 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.632+0000 7f559d8fd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55980fe230 0x7f559810d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:55.634 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.632+0000 7f559d8fd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55980feb80 0x7f559810dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:55.634 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.632+0000 7f559d8fd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559810e380 con 0x7f55980feb80 2023-12-16T20:50:55.634 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.632+0000 7f559d8fd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559810ec20 con 0x7f55980fe230 2023-12-16T20:50:55.634 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.632+0000 7f55967fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55980feb80 0x7f559810dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:55.634 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.632+0000 7f5596ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55980fe230 0x7f559810d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:55.634 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.632+0000 7f55967fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55980feb80 0x7f559810dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52918/0 (socket says 172.21.15.52:52918) 2023-12-16T20:50:55.634 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.632+0000 7f55967fc700 1 -- 172.21.15.52:0/4025616664 learned_addr learned my addr 172.21.15.52:0/4025616664 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-12-16T20:50:55.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.633+0000 7f55967fc700 1 -- 172.21.15.52:0/4025616664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55980fe230 msgr2=0x7f559810d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:55.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.633+0000 7f55967fc700 1 --2- 172.21.15.52:0/4025616664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55980fe230 0x7f559810d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.633+0000 7f55967fc700 1 -- 172.21.15.52:0/4025616664 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5588005040 con 0x7f55980feb80 2023-12-16T20:50:55.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.633+0000 7f5596ffd700 1 --2- 172.21.15.52:0/4025616664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55980fe230 0x7f559810d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T20:50:55.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.633+0000 7f55967fc700 1 --2- 172.21.15.52:0/4025616664 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55980feb80 0x7f559810dd80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f558c002a50 tx=0x7f558c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:55.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.633+0000 7f557f7fe700 1 -- 172.21.15.52:0/4025616664 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f558c007cc0 con 0x7f55980feb80 2023-12-16T20:50:55.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.633+0000 7f559d8fd700 1 -- 172.21.15.52:0/4025616664 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559810ef00 con 0x7f55980feb80 2023-12-16T20:50:55.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.633+0000 7f557f7fe700 1 -- 172.21.15.52:0/4025616664 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f558c00f040 con 0x7f55980feb80 2023-12-16T20:50:55.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.633+0000 7f557f7fe700 1 -- 172.21.15.52:0/4025616664 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f558c00d9c0 con 0x7f55980feb80 2023-12-16T20:50:55.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.633+0000 7f559d8fd700 1 -- 172.21.15.52:0/4025616664 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f559810f440 con 0x7f55980feb80 2023-12-16T20:50:55.636 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.634+0000 7f559d8fd700 1 -- 172.21.15.52:0/4025616664 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55981075e0 con 0x7f55980feb80 2023-12-16T20:50:55.636 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.634+0000 7f557f7fe700 1 -- 172.21.15.52:0/4025616664 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f558c00db20 con 0x7f55980feb80 2023-12-16T20:50:55.636 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.635+0000 7f557f7fe700 1 --2- 172.21.15.52:0/4025616664 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5580068f60 0x7f558006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T20:50:55.636 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.635+0000 7f557f7fe700 1 -- 172.21.15.52:0/4025616664 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f558c086b20 con 0x7f55980feb80 2023-12-16T20:50:55.636 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.635+0000 7f5596ffd700 1 --2- 172.21.15.52:0/4025616664 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5580068f60 0x7f558006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T20:50:55.636 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.635+0000 7f5596ffd700 1 --2- 172.21.15.52:0/4025616664 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5580068f60 0x7f558006b410 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f558800b3b0 tx=0x7f5588009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T20:50:55.640 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.639+0000 7f557f7fe700 1 -- 172.21.15.52:0/4025616664 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f558c0569b0 con 0x7f55980feb80 2023-12-16T20:50:55.852 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.851+0000 7f559d8fd700 1 -- 172.21.15.52:0/4025616664 --> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5598105610 con 0x7f5580068f60 2023-12-16T20:50:55.853 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.852+0000 7f557f7fe700 1 -- 172.21.15.52:0/4025616664 <== mgr.14214 v2:172.21.15.52:6800/2056060258 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7f5598105610 con 0x7f5580068f60 2023-12-16T20:50:55.853 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2023-12-16T20:50:55.853 INFO:teuthology.orchestra.run.smithi052.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5", 2023-12-16T20:50:55.853 INFO:teuthology.orchestra.run.smithi052.stdout: "in_progress": true, 2023-12-16T20:50:55.853 INFO:teuthology.orchestra.run.smithi052.stdout: "which": "Upgrading daemons of type(s) mgr", 2023-12-16T20:50:55.853 INFO:teuthology.orchestra.run.smithi052.stdout: "services_complete": [], 2023-12-16T20:50:55.853 INFO:teuthology.orchestra.run.smithi052.stdout: "progress": "", 2023-12-16T20:50:55.853 INFO:teuthology.orchestra.run.smithi052.stdout: "message": "Currently upgrading mgr daemons", 2023-12-16T20:50:55.854 INFO:teuthology.orchestra.run.smithi052.stdout: "is_paused": false 2023-12-16T20:50:55.854 INFO:teuthology.orchestra.run.smithi052.stdout:} 2023-12-16T20:50:55.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.854+0000 7f559d8fd700 1 -- 172.21.15.52:0/4025616664 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5580068f60 msgr2=0x7f558006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:55.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.854+0000 7f559d8fd700 1 --2- 172.21.15.52:0/4025616664 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5580068f60 0x7f558006b410 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f558800b3b0 tx=0x7f5588009e30 comp rx=0 tx=0).stop 2023-12-16T20:50:55.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.854+0000 7f559d8fd700 1 -- 172.21.15.52:0/4025616664 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55980feb80 msgr2=0x7f559810dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T20:50:55.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.854+0000 7f559d8fd700 1 --2- 172.21.15.52:0/4025616664 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55980feb80 0x7f559810dd80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f558c002a50 tx=0x7f558c0046b0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.855+0000 7f559d8fd700 1 -- 172.21.15.52:0/4025616664 shutdown_connections 2023-12-16T20:50:55.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.855+0000 7f559d8fd700 1 --2- 172.21.15.52:0/4025616664 >> [v2:172.21.15.52:6800/2056060258,v1:172.21.15.52:6801/2056060258] conn(0x7f5580068f60 0x7f558006b410 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.855+0000 7f559d8fd700 1 --2- 172.21.15.52:0/4025616664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55980fe230 0x7f559810d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.855+0000 7f559d8fd700 1 --2- 172.21.15.52:0/4025616664 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55980feb80 0x7f559810dd80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T20:50:55.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.855+0000 7f559d8fd700 1 -- 172.21.15.52:0/4025616664 >> 172.21.15.52:0/4025616664 conn(0x7f55980f69d0 msgr2=0x7f55981013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T20:50:55.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.855+0000 7f559d8fd700 1 -- 172.21.15.52:0/4025616664 shutdown_connections 2023-12-16T20:50:55.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:50:55.855+0000 7f559d8fd700 1 -- 172.21.15.52:0/4025616664 wait complete. 2023-12-16T20:50:56.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:56 smithi052 ceph-mon[80761]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:50:56.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:56 smithi052 ceph-mon[80761]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:50:56.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:56 smithi052 ceph-mon[80761]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T20:50:56.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:50:56 smithi052 ceph-mon[80761]: from='client.? 172.21.15.52:0/1332963085' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T20:50:56.275 INFO:tasks.workunit.client.0.smithi052.stderr:+ tar xzf ltp-full.tgz 2023-12-16T20:51:17.589 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:17.279+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:17.589 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:17.279+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:17.590 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:17.279+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:18.555 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:18.297+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:18.555 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:18.297+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:18.555 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:18.297+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:18.554+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:19.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:18.554+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:19.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:18.554+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:19.338 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:19.009+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:19.338 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:19.009+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:19.338 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:19.009+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:19.337+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:19.337+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:19.337+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:19.523+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:19.523+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:19.523+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:20.329 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:20.053+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:20.329 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:20.053+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:20.329 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:20.053+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:20.328+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:20.328+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:20.328+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:20.487+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:20.487+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:20.487+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:21.348 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:21.080+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:21.348 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:21.080+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:21.348 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:21.080+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:21.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:21.347+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:21.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:21.347+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:21.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:21.347+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:21.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:21.505+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:21.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:21.505+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:21.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:21.505+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:22.329 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:22.077+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:22.329 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:22.077+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:22.329 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:22.077+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:22.590 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:22.328+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:22.590 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:22.328+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:22.590 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:22.328+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:22.590 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:22.485+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:22.590 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:22.485+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:22.590 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:22.485+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:23.341 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:23.057+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:23.341 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:23.057+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:23.341 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:23.057+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:23.340+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:23.340+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:23.340+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:23.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:23.442+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:23.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:23.442+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:23.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:23.442+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:24.449 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:24.324+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:24.449 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:24.324+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:24.449 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:24.324+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:24.449 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:24.095+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:24.449 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:24.095+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:24.449 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:24.095+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:24.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:24.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:24.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:24.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:24.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:24.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:25.330 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:25.066+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:25.330 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:25.066+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:25.330 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:25.066+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:25.329+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:25.329+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:25.329+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:25.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:25.423+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:25.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:25.423+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:25.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:25.423+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:26.455 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:26.052+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:26.455 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:26.052+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:26.456 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:26.052+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:26.456 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:26.280+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:26.456 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:26.280+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:26.456 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:26.280+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:26.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:26.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:26.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:26.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:26.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:26.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:27.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:27.078+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:27.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:27.078+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:27.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:27.078+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:27.589 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:27.255+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:27.590 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:27.255+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:27.590 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:27.255+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:27.590 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:27.444+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:27.590 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:27.444+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:27.590 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:27.444+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:28.450 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:28.127+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:28.450 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:28.127+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:28.450 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:28.127+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:28.450 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:28.223+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:28.451 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:28.223+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:28.451 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:28.223+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:28.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:28.449+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:28.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:28.449+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:28.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:28.449+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:29.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:29.097+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:29.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:29.097+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:29.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:29.097+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:29.417 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:29.198+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:29.418 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:29.198+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:29.418 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:29.198+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:29.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:29.415+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:29.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:29.415+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:29.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:29.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:30.404 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:30.078+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:30.404 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:30.078+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:30.405 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:30.078+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:30.405 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:30.232+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:30.405 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:30.232+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:30.405 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:30.232+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:30.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:30.403+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:30.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:30.403+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:30.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:30.403+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:31.375 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:31.274+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:31.375 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:31.274+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:31.375 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:31.274+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:31.375 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:31.073+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:31.375 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:31.073+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:31.375 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:31.073+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:31.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:31.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:31.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:31.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:31.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:31.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:32.340 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:32.074+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:32.340 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:32.074+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:32.340 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:32.074+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:32.340 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:32.264+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:32.341 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:32.264+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:32.341 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:32.264+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:32.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:32.339+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:32.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:32.339+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:32.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:32.339+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:33.119+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:33.119+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:33.119+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:33.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:33.284+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:33.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:33.284+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:33.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:33.284+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:33.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:33.305+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:33.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:33.305+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:33.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:33.305+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:34.101+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:34.101+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:34.101+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:34.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:34.314+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:34.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:34.314+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:34.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:34.314+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:34.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:34.330+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:34.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:34.330+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:34.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:34.330+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:35.149+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:35.149+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:35.149+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:35.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:35.285+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:35.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:35.285+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:35.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:35.285+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:35.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:35.337+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:35.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:35.337+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:35.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:35.337+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:36.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:36.197+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:36.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:36.197+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:36.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:36.197+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:36.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:36.252+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:36.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:36.252+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:36.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:36.252+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:36.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:36.365+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:36.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:36.365+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:36.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:36.365+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:37.479 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:37.214+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:37.479 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:37.214+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:37.479 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:37.214+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:37.480 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:37.289+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:37.480 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:37.289+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:37.480 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:37.289+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:37.480 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:37.351+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:37.480 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:37.351+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:37.480 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:37.351+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:38.306 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:38.230+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:38.306 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:38.230+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:38.306 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:38.230+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:38.312+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:38.312+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:38.312+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:38.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:38.305+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:38.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:38.305+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:38.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:38.305+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:39.213+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:39.213+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:39.213+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:39.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:39.262+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:39.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:39.262+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:39.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:39.262+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:39.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:39.333+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:39.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:39.333+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:39.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:39.333+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:40.258+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:40.258+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:40.258+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:40.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:40.292+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:40.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:40.292+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:40.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:40.292+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:40.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:40.314+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:40.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:40.314+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:40.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:40.314+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:41.247+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:41.247+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:41.247+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:41.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:41.263+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:41.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:41.263+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:41.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:41.263+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:41.282+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:41.282+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:41.282+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:42.491 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:42.295+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:42.491 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:42.295+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:42.491 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:42.295+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:42.491 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:42.239+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:42.492 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:42.239+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:42.492 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:42.239+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:42.492 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:42.271+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:42.492 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:42.271+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:42.492 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:42.271+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:43.261+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:43.261+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:43.261+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:43.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:43.250+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:43.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:43.250+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:43.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:43.250+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:43.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:43.290+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:43.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:43.290+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:43.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:43.290+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:44.226+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:44.226+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:44.226+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:44.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:44.290+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:44.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:44.290+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:44.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:44.290+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:44.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:44.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:44.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:44.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:44.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:44.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:45.231+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:45.231+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:45.231+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:45.254+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:45.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:45.254+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:45.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:45.254+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:45.322+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:45.322+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:45.322+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:46.284+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:46.284+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:46.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:46.284+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:46.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:46.282+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:46.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:46.282+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:46.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:46.282+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:46.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:46.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:46.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:46.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:46.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:46.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:47.468 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:47.283+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:47.468 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:47.283+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:47.468 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:47.283+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:47.468 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:47.287+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:47.468 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:47.287+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:47.468 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:47.287+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:47.469 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:47.174+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:47.469 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:47.174+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:47.469 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:47.174+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:48.284 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:48.202+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:48.284 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:48.202+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:48.284 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:48.202+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:48.284+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:48.284+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:48.284+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:48.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:48.284+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:48.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:48.284+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:48.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:48.284+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:49.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:49.268+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:49.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:49.268+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:49.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:49.268+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:49.291+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:49.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:49.291+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:49.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:49.291+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:49.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:49.203+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:49.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:49.203+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:49.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:49.203+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:50.247+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:50.247+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:50.247+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:50.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:50.241+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:50.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:50.241+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:50.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:50.241+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:50.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:50.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:50.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:51.225+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:51.225+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:51.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:51.225+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:51.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:51.268+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:51.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:51.268+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:51.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:51.268+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:51.143+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:51.143+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:51.143+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:52.482 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:52.223+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:52.482 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:52.223+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:52.482 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:52.223+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:52.482 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:52.233+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:52.482 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:52.233+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:52.482 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:52.233+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:52.482 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:52.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:52.483 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:52.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:52.483 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:52.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:53.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:53.237+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:53.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:53.237+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:53.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:53.237+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:53.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:53.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:53.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:53.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:53.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:53.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:53.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:53.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:53.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:53.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:53.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:53.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:54.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:54.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:54.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:54.235+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:54.235+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:54.235+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:54.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:54.188+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:54.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:54.188+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:54.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:54.188+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:55.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:55.218+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:55.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:55.218+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:55.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:55.218+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:55.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:55.194+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:55.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:55.194+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:55.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:55.194+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:55.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:55.144+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:55.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:55.144+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:55.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:55.144+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:56.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:56.147+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:56.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:56.147+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:56.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:56.147+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:56.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:56.223+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:56.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:56.223+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:56.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:56.223+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:56.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:56.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:56.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:56.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:56.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:56.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:57.478 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:57.187+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:57.479 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:57.187+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:57.479 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:57.187+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:57.479 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:57.236+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:57.479 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:57.236+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:57.479 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:57.236+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:57.479 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:57.139+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:57.480 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:57.139+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:57.480 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:57.139+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:58.275 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:58.201+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:58.276 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:58.202+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:58.276 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:58.202+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:58.276 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:58.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:58.276 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:58.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:58.276 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:58.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:58.275+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:58.275+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:58.275+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:59.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:59.289+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:59.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:59.289+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:59.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:51:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:51:59.289+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:51:59.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:59.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:59.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:59.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:59.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:51:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:51:59.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:51:59.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:59.137+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:59.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:59.137+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:51:59.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:51:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:51:59.137+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:00.259+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:00.259+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:00.259+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:00.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:00.259+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:00.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:00.259+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:00.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:00.259+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:00.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:00.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:00.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:01.276+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:01.276+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:01.276+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:01.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:01.224+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:01.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:01.224+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:01.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:01.224+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:01.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:01.081+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:01.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:01.081+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:01.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:01.081+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:02.482 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:02.302+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:02.482 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:02.302+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:02.482 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:02.302+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:02.482 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:02.216+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:02.482 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:02.216+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:02.482 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:02.216+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:02.483 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:02.070+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:02.483 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:02.070+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:02.483 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:02.070+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:03.328 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:03.052+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:03.329 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:03.052+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:03.329 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:03.052+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:03.329 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:03.249+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:03.329 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:03.249+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:03.329 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:03.249+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:03.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:03.327+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:03.791 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:03.327+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:03.791 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:03.327+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:04.315+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:04.315+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:04.315+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:04.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:04.275+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:04.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:04.275+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:04.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:04.275+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:04.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:04.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:04.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:05.319 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:05.050+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:05.319 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:05.050+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:05.319 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:05.050+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:05.319 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:05.282+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:05.320 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:05.282+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:05.320 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:05.282+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:05.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:05.318+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:05.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:05.318+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:05.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:05.318+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:06.365 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:06.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:06.366 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:06.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:06.366 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:06.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:06.366 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:06.304+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:06.366 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:06.304+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:06.366 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:06.304+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:06.363+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:06.363+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:06.363+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:07.406 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:07.061+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:07.407 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:07.061+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:07.407 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:07.061+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:07.407 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:07.306+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:07.407 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:07.306+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:07.407 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:07.306+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:07.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:07.405+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:07.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:07.405+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:07.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:07.405+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:08.298 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:08.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:08.298 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:08.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:08.298 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:08.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:08.296+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:08.296+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:08.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:08.296+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:08.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:08.394+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:08.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:08.394+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:08.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:08.394+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:09.346 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:09.068+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:09.346 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:09.068+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:09.346 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:09.068+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:09.345+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:09.345+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:09.345+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:09.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:09.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:09.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:10.438 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:10.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:10.438 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:10.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:10.438 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:10.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:10.438 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:10.364+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:10.438 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:10.364+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:10.438 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:10.364+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:10.436+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:10.436+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:10.436+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:11.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:11.080+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:11.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:11.080+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:11.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:11.080+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:11.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:11.325+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:11.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:11.325+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:11.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:11.325+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:11.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:11.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:11.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:11.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:11.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:11.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:12.420 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:12.290+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:12.420 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:12.290+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:12.420 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:12.290+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:12.420 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:12.064+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:12.420 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:12.064+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:12.420 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:12.064+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:12.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:12.419+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:12.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:12.419+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:12.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:12.419+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:13.413 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:13.093+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:13.413 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:13.093+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:13.413 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:13.093+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:13.414 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:13.256+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:13.414 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:13.256+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:13.414 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:13.256+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:13.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:13.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:13.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:14.421 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:14.065+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:14.421 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:14.065+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:14.421 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:14.065+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:14.421 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:14.239+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:14.421 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:14.239+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:14.421 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:14.239+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:14.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:14.420+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:14.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:14.420+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:14.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:14.420+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:15.441 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:15.087+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:15.442 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:15.087+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:15.442 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:15.087+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:15.442 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:15.201+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:15.442 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:15.201+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:15.442 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:15.201+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:15.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:15.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:15.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:16.463 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:16.106+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:16.463 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:16.106+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:16.463 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:16.106+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:16.463 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:16.226+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:16.463 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:16.226+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:16.463 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:16.226+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:16.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:16.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:16.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:17.440 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:17.264+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:17.440 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:17.264+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:17.440 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:17.264+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:17.440 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:17.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:17.440 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:17.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:17.441 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:17.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:17.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:17.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:17.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:17.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:17.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:17.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:18.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:18.056+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:18.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:18.056+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:18.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:18.056+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:18.466 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:18.248+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:18.466 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:18.248+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:18.466 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:18.248+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:18.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:18.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:18.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:18.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:18.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:18.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:19.466 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:19.048+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:19.466 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:19.048+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:19.466 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:19.048+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:19.467 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:19.290+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:19.467 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:19.290+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:19.467 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:19.291+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:19.465+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:19.465+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:19.465+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:20.418 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:20.085+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:20.418 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:20.085+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:20.418 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:20.086+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:20.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:20.258+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:20.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:20.258+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:20.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:20.258+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:20.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:20.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:20.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:21.442 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:21.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:21.442 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:21.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:21.442 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:21.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:21.442 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:21.260+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:21.442 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:21.260+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:21.442 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:21.260+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:21.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:21.441+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:21.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:21.441+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:21.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:21.441+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:22.464 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:22.242+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:22.464 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:22.242+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:22.465 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:22.242+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:22.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:22.063+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:22.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:22.063+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:22.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:22.063+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:22.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:22.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:22.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:23.478 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:23.060+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:23.478 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:23.060+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:23.478 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:23.060+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:23.478 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:23.271+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:23.478 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:23.271+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:23.479 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:23.271+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:23.477+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:23.477+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:23.477+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:24.475 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:24.102+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:24.475 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:24.102+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:24.475 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:24.102+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:24.475 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:24.260+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:24.475 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:24.260+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:24.475 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:24.260+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:24.474+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:24.474+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:24.474+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:25.430 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:25.054+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:25.430 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:25.054+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:25.430 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:25.054+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:25.431 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:25.276+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:25.431 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:25.276+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:25.431 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:25.276+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:25.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:25.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:25.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:26.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:26.011+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:26.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:26.011+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:26.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:26.011+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:26.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:26.284+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:26.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:26.284+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:26.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:26.284+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:26.475+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:26.475+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:26.475+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:27.474 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:27.237+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:27.474 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:27.237+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:27.474 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:27.237+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:27.475 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:27.025+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:27.475 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:27.025+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:27.475 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:27.025+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:27.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:27.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:27.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:27.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:27.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:27.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:28.256 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:27.999+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:28.256 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:27.999+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:28.256 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:27.999+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:28.257 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:28.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:28.257 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:28.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:28.257 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:28.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:28.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:28.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:28.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:28.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:28.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:28.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:29.246 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:28.958+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:29.246 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:28.958+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:29.246 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:28.958+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:29.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:29.399+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:29.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:29.399+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:29.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:29.399+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:29.245+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:29.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:29.245+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:29.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:29.245+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:30.000+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:30.000+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:30.000+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:30.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:30.216+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:30.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:30.216+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:30.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:30.216+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:30.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:30.416+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:30.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:30.416+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:30.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:30.416+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:31.371 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:31.010+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:31.371 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:31.010+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:31.371 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:31.010+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:31.371 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:31.176+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:31.371 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:31.176+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:31.371 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:31.176+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:31.629 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:31.370+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:31.629 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:31.370+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:31.629 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:31.370+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:32.134+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:32.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:32.134+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:32.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:32.134+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:32.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:31.965+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:32.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:31.965+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:32.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:31.965+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:32.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:32.364+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:32.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:32.364+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:32.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:32.364+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:33.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:32.955+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:33.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:32.955+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:33.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:32.955+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:33.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:33.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:33.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:33.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:33.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:33.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:33.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:33.409+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:33.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:33.409+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:33.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:33.409+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:34.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:33.984+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:34.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:33.984+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:34.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:33.984+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:34.120+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:34.120+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:34.120+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:34.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:34.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:34.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:35.079+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:35.079+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:35.079+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:35.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:34.964+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:35.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:34.964+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:35.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:34.964+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:35.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:35.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:35.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:35.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:35.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:35.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:36.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:35.982+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:36.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:35.982+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:36.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:35.982+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:36.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:36.048+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:36.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:36.048+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:36.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:36.048+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:36.445+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:36.445+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:36.445+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:37.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:36.979+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:37.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:36.979+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:37.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:36.979+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:37.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:37.052+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:37.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:37.052+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:37.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:37.052+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:37.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:37.397+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:37.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:37.397+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:37.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:37.398+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:38.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:38.058+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:38.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:38.058+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:38.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:38.058+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:38.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:37.943+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:38.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:37.943+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:38.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:37.943+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:38.409+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:38.409+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:38.409+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:39.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:38.928+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:39.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:38.928+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:39.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:38.928+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:39.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:39.085+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:39.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:39.085+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:39.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:39.085+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:39.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:39.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:39.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:40.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:39.917+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:40.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:39.917+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:40.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:39.917+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:40.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:40.120+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:40.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:40.120+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:40.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:40.120+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:40.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:40.455+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:40.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:40.455+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:40.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:40.455+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:41.171 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:40.888+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:41.171 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:40.888+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:41.171 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:40.888+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:41.481 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:41.170+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:41.481 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:41.170+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:41.481 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:41.170+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:41.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:41.480+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:41.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:41.480+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:41.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:41.480+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:42.160 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:41.852+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:42.160 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:41.852+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:42.160 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:41.852+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:42.460 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:42.159+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:42.460 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:42.159+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:42.460 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:42.159+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:42.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:42.459+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:42.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:42.459+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:42.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:42.459+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:43.134 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:42.844+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:43.134 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:42.844+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:43.134 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:42.844+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:43.451 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:43.133+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:43.452 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:43.133+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:43.452 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:43.133+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:43.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:43.451+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:43.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:43.451+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:43.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:43.451+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:44.135 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:43.805+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:44.135 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:43.805+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:44.135 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:43.805+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:44.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:44.134+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:44.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:44.134+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:44.419 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:44.134+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:44.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:44.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:44.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:44.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:44.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:44.418+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:45.178 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:44.851+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:45.178 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:44.851+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:45.178 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:44.851+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:45.176+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:45.176+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:45.176+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:45.373+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:45.373+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:45.373+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:46.198 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:45.857+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:46.198 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:45.857+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:46.198 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:45.857+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:46.196+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:46.196+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:46.196+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:46.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:46.351+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:46.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:46.351+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:46.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:46.351+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:47.232 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:46.881+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:47.233 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:46.881+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:47.233 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:46.881+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:47.484 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:47.313+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:47.485 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:47.313+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:47.485 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:47.313+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:47.485 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:47.231+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:47.485 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:47.231+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:47.485 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:47.231+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:48.210 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:47.857+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:48.211 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:47.857+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:48.211 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:47.857+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:48.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:48.210+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:48.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:48.210+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:48.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:48.210+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:48.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:48.268+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:48.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:48.268+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:48.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:48.268+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:49.195 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:48.872+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:49.195 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:48.872+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:49.195 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:48.872+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:49.194+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:49.194+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:49.194+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:49.251+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:49.251+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:49.251+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:50.220 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:49.829+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:50.220 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:49.829+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:50.220 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:49.829+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:50.219+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:50.219+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:50.219+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:50.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:50.281+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:50.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:50.281+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:50.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:50.281+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:51.183 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:50.866+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:51.183 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:50.866+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:51.183 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:50.866+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:51.182+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:51.182+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:51.182+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:51.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:51.252+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:51.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:51.252+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:51.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:51.252+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:51.832+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:51.832+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:51.832+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:52.486 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:52.211+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:52.486 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:52.211+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:52.486 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:52.211+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:52.486 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:52.203+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:52.486 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:52.203+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:52.486 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:52.203+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:53.181 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:52.813+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:53.181 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:52.813+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:53.181 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:52.813+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:53.492 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:53.181+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:53.492 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:53.181+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:53.492 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:53.181+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:53.492 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:53.250+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:53.492 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:53.250+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:53.492 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:53.250+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:54.136 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:53.830+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:54.137 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:53.830+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:54.137 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:53.830+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:54.273+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:54.273+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:54.273+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:54.136+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:54.136+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:54.136+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:55.225 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:55.087+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:55.225 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:55.087+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:55.225 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:55.087+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:55.226 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:54.872+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:55.226 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:54.872+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:55.226 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:54.872+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:55.225+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:55.225+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:55.225+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:56.181 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:56.072+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:56.181 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:56.072+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:56.181 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:56.072+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:56.181 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:55.884+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:56.182 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:55.884+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:56.182 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:55.884+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:56.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:56.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:56.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:56.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:56.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:56.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:57.170 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:57.037+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:57.170 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:57.037+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:57.170 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:57.037+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:57.171 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:56.903+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:57.171 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:56.903+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:57.171 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:56.903+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:57.487 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:57.169+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:57.488 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:57.169+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:57.488 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:57.169+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:58.188 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:58.071+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:58.189 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:58.071+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:58.189 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:58.071+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:58.189 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:57.925+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:58.189 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:57.925+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:58.189 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:57.925+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:58.187+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:58.187+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:58.187+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:59.198 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:59.119+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:59.199 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:59.119+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:59.199 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:52:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:52:59.119+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:52:59.199 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:58.909+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:59.199 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:58.909+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:59.199 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:58.909+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:52:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:59.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:59.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:52:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:52:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:52:59.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:00.210 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:00.160+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:00.210 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:00.160+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:00.210 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:00.160+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:00.211 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:59.949+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:00.211 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:59.949+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:00.211 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:52:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:52:59.949+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:00.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:00.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:00.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:01.204 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:00.954+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:01.204 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:00.954+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:01.204 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:00.954+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:01.210+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:01.210+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:01.210+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:01.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:01.203+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:01.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:01.203+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:01.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:01.203+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:02.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:02.184+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:02.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:02.184+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:02.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:02.184+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:02.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:01.981+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:02.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:01.981+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:02.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:01.981+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:02.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:02.155+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:02.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:02.155+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:02.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:02.155+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:02.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:53:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:53:02.475+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:53:03.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:03.167+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:03.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:03.167+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:03.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:03.167+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:03.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:03.113+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:03.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:03.113+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:03.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:03.113+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:03.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:02.961+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:03.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:02.961+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:03.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:02.961+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:04.135+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:04.135+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:04.135+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:04.151+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:04.152+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:04.152+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:04.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:03.991+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:04.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:03.991+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:04.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:03.991+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:05.086+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:05.086+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:05.086+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:05.119+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:05.119+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:05.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:05.119+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:05.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:04.945+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:05.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:04.945+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:05.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:04.945+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:06.065 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:05.956+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:06.065 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:05.956+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:06.065 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:05.956+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:06.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:06.144+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:06.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:06.144+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:06.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:06.144+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:06.064+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:06.064+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:06.064+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:07.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:07.034+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:07.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:07.034+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:07.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:07.034+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:07.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:07.110+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:07.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:07.110+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:07.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:07.110+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:07.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:06.949+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:07.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:06.949+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:07.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:06.949+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:07.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:53:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:53:07.476+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 17 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:53:08.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:08.077+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:08.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:08.077+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:08.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:08.077+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:08.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:07.961+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:08.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:07.961+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:08.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:07.961+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:08.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:08.125+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:08.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:08.125+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:08.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:08.125+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:09.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:09.135+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:09.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:09.135+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:09.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:09.135+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:09.095+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:09.095+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:09.095+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:09.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:08.994+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:09.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:08.994+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:09.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:08.994+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:10.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:10.165+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:10.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:10.165+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:10.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:10.165+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:10.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:10.061+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:10.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:10.061+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:10.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:10.061+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:10.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:09.970+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:10.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:09.970+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:10.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:09.970+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:11.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:11.196+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:11.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:11.196+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:11.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:11.196+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:11.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:11.013+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:11.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:11.013+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:11.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:11.013+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:11.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:11.009+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:11.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:11.009+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:11.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:11.009+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:12.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:12.022+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:12.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:12.022+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:12.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:12.022+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:12.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:12.212+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:12.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:12.212+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:12.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:12.212+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:12.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:11.978+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:12.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:11.978+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:12.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:11.978+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:12.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:53:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:53:12.476+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 26 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:53:13.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:13.014+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:13.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:13.014+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:13.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:13.014+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:13.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:12.960+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:13.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:12.960+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:13.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:12.960+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:13.520 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:13.256+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:13.521 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:13.256+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:13.521 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:13.256+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:14.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:14.021+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:14.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:14.021+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:14.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:14.021+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:14.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:13.915+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:14.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:13.915+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:14.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:13.915+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:14.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:14.295+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:14.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:14.295+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:14.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:14.295+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:14.987+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:14.987+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:14.987+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:15.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:14.924+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:15.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:14.924+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:15.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:14.924+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:15.326+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:15.326+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:15.326+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:15.941+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:15.941+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:15.941+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:16.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:15.900+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:16.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:15.900+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:16.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:15.900+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:16.358+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:16.358+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:16.358+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:17.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:16.944+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:17.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:16.944+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:17.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:16.944+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:17.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:16.907+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:17.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:16.907+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:17.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:16.907+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:17.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:53:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:53:17.477+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 47 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:53:17.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:17.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:17.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:17.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:17.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:17.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:18.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:17.977+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:18.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:17.977+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:18.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:17.977+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:18.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:17.956+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:18.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:17.956+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:18.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:17.956+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:18.422+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:18.422+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:18.422+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:19.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:18.971+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:19.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:18.971+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:19.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:18.971+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:19.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:18.913+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:19.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:18.913+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:19.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:18.913+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:19.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:19.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:19.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:20.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:19.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:20.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:19.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:20.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:19.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:20.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:19.929+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:20.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:19.929+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:20.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:19.929+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:20.438+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:20.438+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:20.438+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:21.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:20.985+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:21.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:20.985+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:21.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:20.985+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:21.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:20.942+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:21.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:20.942+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:21.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:20.942+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:21.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:21.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:21.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:21.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:21.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:21.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:22.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:22.009+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:22.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:22.009+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:22.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:22.009+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:22.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:21.943+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:22.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:21.943+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:22.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:21.943+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:22.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:53:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:53:22.477+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 78 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:53:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:22.441+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:22.441+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:22.441+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:23.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:22.952+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:23.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:22.952+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:23.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:22.952+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:23.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:23.050+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:23.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:23.050+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:23.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:23.050+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:23.434+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:23.434+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:23.434+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:24.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:23.945+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:24.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:23.945+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:24.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:23.945+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:24.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:24.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:24.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:24.476+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:24.476+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:24.476+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:25.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:24.954+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:25.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:24.954+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:25.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:24.954+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:25.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:25.016+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:25.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:25.016+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:25.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:25.016+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:25.441+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:25.441+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:25.441+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:26.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:26.057+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:26.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:26.057+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:26.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:26.057+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:26.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:25.954+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:26.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:25.954+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:26.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:25.954+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:26.414+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:26.414+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:26.414+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:27.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:26.929+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:27.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:26.929+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:27.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:26.929+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:27.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:27.062+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:27.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:27.062+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:27.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:27.062+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:27.607 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:53:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:53:27.477+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 105 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:53:27.607 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:27.456+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:27.607 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:27.456+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:27.607 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:27.456+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:28.014+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:28.014+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:28.014+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:27.949+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:27.949+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:27.949+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:28.416+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:28.416+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:28.416+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:29.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:28.914+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:29.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:28.914+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:29.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:28.914+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:29.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:28.984+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:29.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:28.984+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:29.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:28.984+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:29.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:29.373+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:29.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:29.373+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:29.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:29.373+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:30.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:30.013+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:30.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:30.013+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:30.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:30.013+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:29.941+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:29.941+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:29.941+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:30.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:30.369+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:30.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:30.369+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:30.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:30.369+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:31.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:31.010+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:31.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:31.010+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:31.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:31.010+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:31.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:30.916+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:31.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:30.916+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:31.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:30.916+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:31.633 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:31.377+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:31.633 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:31.377+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:31.633 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:31.377+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:32.060+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:32.060+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:32.060+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:32.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:31.919+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:32.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:31.919+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:32.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:31.919+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:32.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:53:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:53:32.478+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 135 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:53:32.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:32.358+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:32.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:32.358+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:32.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:32.358+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:33.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:33.019+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:33.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:33.019+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:33.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:33.019+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:33.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:32.938+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:33.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:32.938+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:33.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:32.938+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:33.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:33.354+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:33.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:33.354+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:33.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:33.354+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:34.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:33.928+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:34.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:33.928+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:34.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:33.928+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:34.027+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:34.027+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:34.027+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:34.367+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:34.367+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:34.367+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:35.066+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:35.066+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:35.066+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:35.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:34.965+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:35.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:34.965+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:35.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:34.965+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:35.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:35.363+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:35.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:35.363+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:35.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:35.363+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:36.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:36.037+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:36.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:36.037+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:36.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:36.037+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:36.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:35.947+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:36.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:35.947+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:36.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:35.947+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:36.360+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:36.360+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:36.360+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:37.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:36.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:37.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:36.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:37.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:36.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:37.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:36.905+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:37.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:36.905+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:37.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:36.905+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:37.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:53:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:53:37.479+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 163 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:53:37.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:37.402+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:37.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:37.402+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:37.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:37.402+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:38.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:37.941+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:38.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:37.941+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:38.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:37.941+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:38.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:37.879+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:38.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:37.879+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:38.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:37.879+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:38.409+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:38.409+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:38.409+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:39.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:38.983+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:39.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:38.983+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:39.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:38.983+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:39.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:38.832+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:39.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:38.832+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:39.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:38.832+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:39.372+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:39.372+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:39.372+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:40.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:40.023+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:40.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:40.023+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:40.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:40.023+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:40.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:39.832+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:40.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:39.832+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:40.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:39.832+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:40.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:40.397+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:40.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:40.397+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:40.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:40.397+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:41.047 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:40.793+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:41.047 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:40.793+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:41.047 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:40.793+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:41.356 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:41.046+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:41.356 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:41.046+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:41.356 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:41.046+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:41.634 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:41.355+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:41.634 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:41.356+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:41.634 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:41.356+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:42.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:41.803+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:42.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:41.803+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:42.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:41.803+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:42.395 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:42.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:42.395 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:42.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:42.395 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:42.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:42.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:53:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:53:42.480+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 194 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:53:42.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:42.394+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:42.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:42.394+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:42.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:42.394+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:43.125 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:42.791+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:43.125 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:42.791+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:43.125 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:42.791+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:43.400 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:43.124+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:43.400 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:43.124+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:43.401 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:43.124+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:43.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:43.400+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:43.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:43.400+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:43.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:43.400+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:44.139 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:43.776+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:44.139 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:43.776+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:44.139 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:43.776+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:44.411 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:44.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:44.412 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:44.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:44.412 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:44.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:44.739 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:44.411+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:44.739 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:44.411+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:44.739 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:44.411+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:45.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:44.738+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:45.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:44.738+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:45.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:44.738+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:45.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:45.161+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:45.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:45.161+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:45.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:45.161+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:45.716 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:45.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:45.716 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:45.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:45.716 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:45.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:46.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:45.732+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:46.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:45.732+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:46.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:45.732+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:46.422 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:46.153+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:46.422 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:46.153+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:46.422 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:46.153+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:46.744 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:46.421+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:46.744 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:46.421+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:46.744 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:46.421+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:47.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:46.743+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:47.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:46.744+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:47.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:46.744+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:47.396 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:47.104+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:47.396 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:47.104+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:47.396 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:47.104+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:47.742 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:53:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:53:47.481+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 221 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:53:47.742 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:47.395+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:47.742 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:47.395+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:47.742 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:47.395+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:48.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:47.741+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:48.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:47.741+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:48.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:47.741+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:48.414 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:48.130+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:48.414 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:48.130+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:48.414 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:48.130+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:48.742 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:48.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:48.743 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:48.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:48.743 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:48.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:49.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:48.742+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:49.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:48.742+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:49.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:48.742+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:49.136+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:49.136+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:49.136+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:49.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:49.368+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:49.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:49.368+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:49.368+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:50.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:49.738+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:50.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:49.738+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:50.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:49.738+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:50.111+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:50.111+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:50.111+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:50.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:50.336+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:50.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:50.336+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:50.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:50.336+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:51.130 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:50.782+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:51.130 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:50.782+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:51.130 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:50.782+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:51.382 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:51.129+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:51.382 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:51.129+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:51.383 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:51.129+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:51.634 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:51.382+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:51.635 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:51.382+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:51.635 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:51.382+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:51.794+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:51.794+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:51.794+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:52.401 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:52.110+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:52.401 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:52.110+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:52.402 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:52.110+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:52.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:53:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:53:52.482+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 251 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:53:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:52.400+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:52.400+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:52.400+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:53.088+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:53.088+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:53.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:53.088+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:53.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:52.840+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:53.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:52.840+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:53.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:52.840+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:53.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:53.385+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:53.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:53.385+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:53.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:53.385+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:54.111 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:53.856+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:54.111 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:53.856+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:54.111 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:53.856+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:54.411 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:54.110+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:54.412 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:54.110+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:54.412 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:54.110+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:54.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:54.411+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:54.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:54.411+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:54.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:54.411+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:55.082 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:54.807+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:55.082 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:54.807+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:55.082 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:54.807+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:55.440 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:55.081+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:55.440 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:55.081+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:55.440 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:55.081+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:55.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:55.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:55.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:55.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:55.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:55.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:56.076+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:56.076+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:56.076+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:56.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:55.838+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:56.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:55.838+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:56.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:55.838+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:56.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:56.406+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:56.406+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:56.406+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:57.118 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:56.835+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:57.119 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:56.835+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:57.119 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:56.835+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:57.401 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:57.118+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:57.401 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:57.118+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:57.401 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:57.118+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:57.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:53:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:53:57.482+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 279 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:53:57.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:57.401+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:57.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:57.401+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:57.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:57.401+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:58.167 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:57.794+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:58.167 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:57.794+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:58.167 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:57.794+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:58.166+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:58.166+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:58.166+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:58.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:58.379+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:58.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:58.379+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:58.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:58.379+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:59.158 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:58.814+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:59.159 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:58.814+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:59.159 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:58.814+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:53:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:59.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:59.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:53:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:53:59.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:53:59.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:59.363+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:59.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:59.363+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:53:59.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:53:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:53:59.363+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:00.195 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:59.794+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:00.196 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:59.794+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:00.196 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:53:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:53:59.794+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:00.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:00.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:00.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:00.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:00.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:00.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:00.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:00.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:00.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:01.076 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:00.779+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:01.076 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:00.779+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:01.076 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:00.779+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:01.364 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:01.182+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:01.364 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:01.182+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:01.364 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:01.182+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:01.364 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:01.183+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:01.637 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:01.363+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:01.638 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:01.363+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:01.638 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:01.363+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:01.638 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:01.364+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:02.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:01.800+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:02.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:01.800+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:02.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:01.800+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:02.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:01.801+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:02.484 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:02.316+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:02.484 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:02.316+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:02.484 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:02.316+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:02.484 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:02.317+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:02.484 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:02.196+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:02.485 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:02.196+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:02.485 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:02.196+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:02.485 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:02.196+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:02.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:54:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:54:02.483+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 310 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:54:03.159 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:02.763+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:03.159 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:02.763+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:03.159 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:02.763+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:03.159 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:02.764+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:03.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:03.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:03.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:03.159+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:03.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:03.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:03.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:03.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:03.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:03.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:03.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:03.339+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:04.144 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:03.785+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:04.144 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:03.785+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:04.144 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:03.785+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:04.144 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:03.786+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:04.143+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:04.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:04.143+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:04.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:04.143+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:04.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:04.144+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:04.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:04.345+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:04.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:04.345+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:04.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:04.345+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:04.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:04.346+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:05.141 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:04.774+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:05.141 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:04.774+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:05.141 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:04.774+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:05.141 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:04.774+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:05.141+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:05.141+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:05.141+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:05.141+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:05.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:05.304+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:05.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:05.304+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:05.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:05.304+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:05.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:05.304+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:06.143 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:05.755+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:06.144 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:05.755+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:06.144 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:05.755+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:06.144 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:05.755+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:06.143+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:06.143+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:06.143+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:06.143+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:06.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:06.314+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:06.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:06.314+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:06.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:06.314+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:06.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:06.314+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:07.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:06.706+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:07.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:06.706+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:07.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:06.706+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:07.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:06.706+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:07.485 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:07.355+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:07.485 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:07.355+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:07.485 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:07.355+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:07.485 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:07.355+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:07.485 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:07.177+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:07.485 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:07.177+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:07.485 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:07.177+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:07.485 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:07.177+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:07.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:54:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:54:07.484+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 337 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:54:07.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:07.707+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:07.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:07.707+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:07.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:07.707+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:07.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:07.707+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:08.323 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:08.148+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:08.323 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:08.148+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:08.323 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:08.148+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:08.323 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:08.148+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:08.677 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:08.322+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:08.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:08.322+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:08.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:08.322+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:08.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:08.323+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:09.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:08.677+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:09.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:08.677+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:09.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:08.677+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:09.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:08.677+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:09.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:09.372+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:09.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:09.372+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:09.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:09.372+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:09.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:09.373+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:09.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:09.189+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:09.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:09.189+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:09.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:09.189+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:09.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:09.190+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:10.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:09.657+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:10.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:09.657+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:10.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:09.657+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:10.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:09.657+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:10.197+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:10.197+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:10.197+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:10.197+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:10.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:10.410+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:10.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:10.410+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:10.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:10.410+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:10.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:10.411+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:11.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:10.618+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:11.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:10.618+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:11.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:10.618+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:11.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:10.618+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:11.434 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:11.151+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:11.434 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:11.151+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:11.434 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:11.151+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:11.434 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:11.151+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:11.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:11.434+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:11.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:11.434+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:11.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:11.434+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:11.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:11.434+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:11.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:11.588+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:11.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:11.588+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:11.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:11.588+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:11.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:11.588+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:12.486 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:12.424+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:12.486 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:12.424+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:12.486 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:12.424+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:12.486 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:12.425+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:12.486 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:12.196+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:12.486 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:12.196+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:12.487 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:12.196+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:12.487 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:12.197+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:12.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:54:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:54:12.484+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 369 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:54:12.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:12.598+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:12.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:12.598+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:12.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:12.598+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:12.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:12.598+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:13.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:13.424+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:13.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:13.424+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:13.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:13.424+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:13.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:13.424+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:13.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:13.231+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:13.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:13.231+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:13.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:13.231+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:13.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:13.231+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:14.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:13.618+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:14.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:13.618+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:14.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:13.618+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:14.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:13.618+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:14.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:14.416+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:14.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:14.416+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:14.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:14.416+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:14.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:14.416+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:14.202+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:14.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:14.202+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:14.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:14.202+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:14.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:14.203+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:15.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:14.604+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:15.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:14.604+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:15.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:14.604+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:15.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:14.604+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:15.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:15.416+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:15.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:15.416+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:15.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:15.416+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:15.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:15.417+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:15.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:15.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:15.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:15.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:15.217+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:15.575+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:15.575+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:15.575+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:15.575+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:16.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:16.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:16.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:16.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:16.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:16.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:16.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:16.404+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:16.246+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:16.246+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:16.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:16.246+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:16.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:16.246+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:16.558+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:16.558+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:16.558+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:16.558+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:17.568 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:54:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:54:17.485+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 397 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:54:17.568 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:17.295+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:17.568 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:17.295+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:17.568 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:17.295+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:17.568 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:17.295+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:17.568 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:17.438+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:17.568 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:17.438+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:17.569 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:17.438+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:17.569 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:17.438+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:18.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:17.567+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:18.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:17.567+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:18.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:17.567+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:18.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:17.567+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:18.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:18.288+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:18.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:18.288+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:18.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:18.288+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:18.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:18.289+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:18.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:18.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:18.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:18.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:18.404+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:18.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:18.530+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:18.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:18.530+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:18.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:18.530+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:18.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:18.530+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:19.285+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:19.285+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:19.285+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:19.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:19.285+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:19.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:19.385+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:19.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:19.385+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:19.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:19.385+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:19.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:19.385+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:19.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:19.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:19.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:19.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:19.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:19.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:19.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:19.529+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:20.303+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:20.303+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:20.303+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:20.303+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:20.389+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:20.389+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:20.389+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:20.390+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:20.549+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:20.549+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:20.549+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:20.549+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:21.592 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:21.256+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:21.592 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:21.256+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:21.592 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:21.256+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:21.592 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:21.256+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:21.592 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:21.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:21.592 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:21.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:21.593 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:21.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:21.593 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:21.404+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:21.591+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:21.591+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:21.591+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:21.592+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:22.525 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:54:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:54:22.486+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 443 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:54:22.526 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:22.268+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:22.526 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:22.268+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:22.526 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:22.268+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:22.526 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:22.268+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:22.526 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:22.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:22.526 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:22.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:22.526 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:22.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:22.526 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:22.404+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:23.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:22.548+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:23.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:22.548+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:23.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:22.548+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:23.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:22.549+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:23.525 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:23.257+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:23.526 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:23.257+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:23.526 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:23.257+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:23.526 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:23.257+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:23.526 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:23.449+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:23.526 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:23.449+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:23.526 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:23.449+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:23.526 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:23.450+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:24.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:23.581+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:24.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:23.581+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:24.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:23.581+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:24.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:23.582+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:24.536 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:24.274+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:24.536 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:24.274+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:24.536 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:24.274+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:24.536 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:24.274+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:24.536 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:24.469+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:24.537 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:24.469+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:24.537 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:24.469+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:24.537 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:24.469+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:24.536+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:24.536+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:24.536+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:24.536+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:25.572 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:25.285+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:25.572 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:25.285+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:25.572 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:25.285+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:25.572 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:25.285+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:25.572 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:25.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:25.572 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:25.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:25.572 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:25.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:25.572 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:25.484+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:25.571+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:25.571+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:25.571+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:25.571+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:26.462 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:26.248+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:26.463 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:26.248+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:26.463 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:26.248+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:26.463 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:26.248+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:26.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:26.587+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:26.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:26.587+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:26.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:26.587+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:26.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:26.587+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:26.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:26.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:26.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:26.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:26.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:26.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:26.462+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:27.488 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:27.231+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:27.488 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:27.231+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:27.488 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:27.231+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:27.488 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:27.231+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:27.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:54:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:54:27.487+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 472 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:54:27.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:27.601+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:27.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:27.601+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:27.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:27.601+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:27.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:27.601+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:27.487+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:27.487+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:27.487+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:27.488+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:28.499 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:28.219+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:28.499 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:28.219+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:28.500 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:28.219+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:28.500 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:28.219+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:28.499+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:28.499+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:28.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:28.499+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:28.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:28.499+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:28.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:28.608+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:28.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:28.608+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:28.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:28.608+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:28.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:28.608+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:29.197+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:29.197+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:29.197+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:29.197+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:29.509+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:29.509+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:29.509+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:29.509+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:30.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:29.565+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:30.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:29.565+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:30.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:29.565+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:30.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:29.566+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:30.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:30.229+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:30.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:30.229+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:30.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:30.229+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:30.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:30.230+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:30.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:30.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:30.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:30.556+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:31.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:30.523+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:31.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:30.523+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:31.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:30.523+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:31.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:30.523+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:31.238+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:31.238+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:31.238+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:31.238+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:31.796 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:31.504+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:31.796 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:31.504+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:31.797 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:31.504+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:31.797 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:31.505+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:31.797 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:31.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:31.797 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:31.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:31.797 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:31.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:31.797 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:31.556+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:32.516 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:54:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:54:32.487+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 504 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:54:32.516 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:32.462+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:32.516 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:32.462+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:32.516 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:32.462+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:32.516 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:32.462+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:32.517 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:32.261+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:32.517 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:32.261+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:32.517 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:32.261+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:32.517 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:32.262+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:33.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:32.577+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:33.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:32.577+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:33.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:32.577+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:33.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:32.578+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:33.497 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:33.235+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:33.498 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:33.235+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:33.498 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:33.235+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:33.498 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:33.236+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:33.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:33.497+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:33.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:33.497+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:33.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:33.497+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:33.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:33.497+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:33.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:33.581+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:33.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:33.581+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:33.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:33.581+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:33.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:33.581+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:34.202+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:34.202+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:34.202+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:34.202+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:34.506+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:34.506+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:34.506+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:34.506+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:34.535+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:35.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:34.535+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:35.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:34.535+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:35.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:34.535+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:35.154+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:35.154+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:35.154+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:35.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:35.154+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:35.535+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:35.535+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:35.535+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:35.536+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:35.562+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:36.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:35.562+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:36.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:35.562+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:36.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:35.562+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:36.494 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:36.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:36.495 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:36.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:36.495 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:36.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:36.495 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:36.179+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:36.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:36.494+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:36.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:36.494+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:36.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:36.494+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:36.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:36.494+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:36.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:36.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:36.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:36.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:36.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:36.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:36.605+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:37.489 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:37.132+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:37.489 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:37.132+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:37.489 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:37.132+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:37.490 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:37.132+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:37.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:54:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:54:37.488+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 532 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:54:37.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:37.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:37.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:37.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:37.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:37.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:37.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:37.611+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:37.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:37.501+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:37.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:37.501+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:37.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:37.501+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:37.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:37.502+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:38.146+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:38.146+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:38.146+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:38.146+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:39.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:38.507+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:39.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:38.507+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:39.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:38.507+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:39.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:38.507+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:39.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:38.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:39.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:38.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:39.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:38.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:39.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:38.585+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:39.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:39.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:39.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:39.195+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:40.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:39.526+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:40.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:39.526+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:40.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:39.526+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:40.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:39.526+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:39.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:39.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:39.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:39.611+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:40.230+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:40.230+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:40.230+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:40.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:40.231+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:40.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:40.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:40.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:40.563+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:40.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:40.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:40.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:40.611+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:41.203+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:41.203+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:41.203+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:41.204+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:41.654+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:41.654+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:41.654+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:41.654+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:42.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:41.546+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:42.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:41.546+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:42.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:41.546+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:42.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:41.546+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:42.490 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:42.168+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:42.490 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:42.168+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:42.490 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:42.168+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:42.490 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:42.168+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:42.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:54:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:54:42.489+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 563 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:54:42.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:42.616+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:42.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:42.616+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:42.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:42.616+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:42.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:42.616+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:42.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:42.578+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:42.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:42.578+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:42.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:42.578+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:42.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:42.578+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:43.211+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:43.211+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:43.211+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:43.211+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:43.645+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:43.645+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:43.645+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:43.645+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:44.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:43.592+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:44.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:43.592+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:44.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:43.592+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:44.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:43.592+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:44.184+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:44.184+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:44.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:44.184+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:44.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:44.184+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:44.674+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:44.674+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:44.674+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:44.674+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:45.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:44.572+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:45.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:44.573+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:45.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:44.573+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:45.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:44.573+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:45.220+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:45.220+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:45.220+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:45.220+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:45.671+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:45.671+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:45.671+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:45.671+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:46.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:45.572+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:46.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:45.572+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:46.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:45.572+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:46.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:45.572+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:46.177+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:46.177+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:46.177+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:46.177+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:46.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:46.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:47.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:46.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:47.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:46.633+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:47.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:46.571+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:47.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:46.571+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:47.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:46.571+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:47.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:46.572+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:47.491 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:47.186+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:47.491 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:47.186+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:47.491 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:47.186+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:47.491 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:47.186+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:47.491 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:54:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:54:47.489+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 590 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:54:47.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:47.596+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:47.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:47.596+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:47.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:47.596+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:47.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:47.597+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:47.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:47.531+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:47.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:47.531+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:47.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:47.531+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:47.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:47.531+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:48.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:48.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:48.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:48.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:48.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:48.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:48.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:48.180+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:48.597+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:48.597+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:49.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:48.597+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:49.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:48.597+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:49.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:48.556+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:49.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:48.556+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:49.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:48.556+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:49.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:48.557+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:49.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:49.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:49.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:49.178+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:49.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:49.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:49.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:49.551+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:50.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:49.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:50.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:49.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:50.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:49.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:50.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:49.564+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:50.174+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:50.174+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:50.174+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:50.174+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:50.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:50.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:51.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:50.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:51.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:50.572+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:51.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:50.533+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:51.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:50.533+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:51.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:50.533+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:51.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:50.533+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:51.203+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:51.203+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:51.203+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:51.204+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:51.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:51.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:51.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:51.551+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:51.532+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:52.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:51.532+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:52.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:51.532+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:52.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:51.532+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:52.491 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:52.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:52.491 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:52.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:52.491 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:52.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:52.491 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:52.217+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:52.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:54:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:54:52.489+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 620 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:54:52.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:52.591+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:52.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:52.591+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:52.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:52.591+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:52.592+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:52.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:52.502+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:52.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:52.502+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:52.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:52.502+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:52.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:52.503+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:53.172+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:53.172+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:53.172+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:53.172+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:53.510+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:53.510+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:53.510+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:53.510+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:53.575+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:53.575+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:53.575+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:54.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:53.575+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:54.473 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:54.145+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:54.474 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:54.145+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:54.474 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:54.145+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:54.474 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:54.146+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:54.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:54.575+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:54.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:54.575+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:54.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:54.575+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:54.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:54.575+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:54.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:54.472+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:54.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:54.472+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:54.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:54.472+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:54.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:54.472+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:55.479 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:55.112+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:55.479 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:55.112+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:55.479 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:55.112+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:55.479 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:55.112+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:55.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:55.527+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:55.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:55.527+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:55.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:55.527+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:55.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:55.527+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:55.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:55.477+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:55.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:55.477+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:55.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:55.477+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:55.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:55.477+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:56.475 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:56.106+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:56.475 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:56.106+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:56.475 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:56.106+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:56.475 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:56.107+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:56.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:56.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:56.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:56.474+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:56.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:56.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:56.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:56.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:56.527+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:57.427 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:57.068+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:57.428 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:57.068+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:57.428 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:57.068+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:57.428 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:57.068+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:57.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:54:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:54:57.490+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 648 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:54:57.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:57.544+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:57.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:57.544+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:57.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:57.544+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:57.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:57.545+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:57.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:57.426+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:57.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:57.426+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:57.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:57.426+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:57.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:57.426+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:58.446 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:58.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:58.446 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:58.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:58.446 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:58.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:58.446 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:58.082+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:58.501+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:58.501+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:58.501+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:58.501+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:58.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:58.445+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:58.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:58.445+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:58.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:58.445+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:58.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:58.445+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:59.444 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:59.055+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:59.444 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:59.055+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:59.444 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:59.055+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:54:59.444 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:54:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:54:59.055+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:59.482+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:59.482+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:59.482+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:54:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:54:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:54:59.482+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:54:59.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:59.457+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:59.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:59.457+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:59.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:59.457+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:54:59.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:54:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:54:59.457+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:00.411 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:00.052+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:00.411 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:00.052+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:00.412 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:00.052+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:00.412 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:00.052+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:00.476+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:00.476+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:00.476+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:00.476+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:00.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:00.411+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:00.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:00.411+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:00.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:00.411+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:00.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:00.411+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:01.392 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:01.078+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:01.393 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:01.078+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:01.393 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:01.078+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:01.393 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:01.078+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:01.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:01.450+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:01.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:01.450+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:01.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:01.450+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:01.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:01.450+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:01.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:01.392+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:01.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:01.392+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:01.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:01.392+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:01.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:01.392+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:02.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:02.123+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:02.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:02.123+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:02.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:02.123+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:02.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:02.123+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:02.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:55:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:55:02.492+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 679 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:55:02.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:02.446+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:02.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:02.446+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:02.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:02.446+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:02.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:02.447+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:02.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:02.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:02.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:02.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:02.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:02.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:02.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:02.416+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:03.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:03.173+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:03.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:03.173+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:03.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:03.173+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:03.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:03.173+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:03.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:03.416+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:03.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:03.416+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:03.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:03.416+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:03.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:03.417+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:03.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:03.406+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:03.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:03.406+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:03.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:03.406+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:03.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:03.407+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:04.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:04.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:04.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:04.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:04.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:04.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:04.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:04.453+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:04.436+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:04.436+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:04.436+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:04.436+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:04.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:04.222+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:04.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:04.222+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:04.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:04.222+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:04.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:04.223+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:05.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:05.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:05.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:05.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:05.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:05.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:05.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:05.413+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:05.215+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:05.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:05.215+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:05.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:05.215+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:05.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:05.216+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:05.434+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:05.434+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:05.434+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:05.434+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:06.399 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:06.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:06.399 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:06.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:06.400 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:06.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:06.400 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:06.180+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:06.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:06.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:06.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:06.434+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:06.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:06.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:06.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:06.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:06.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:06.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:06.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:06.399+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:07.410 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:07.151+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:07.410 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:07.151+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:07.410 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:07.151+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:07.410 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:07.151+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:07.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:55:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:55:07.493+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 708 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:55:07.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:07.409+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:07.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:07.409+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:07.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:07.409+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:07.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:07.409+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:07.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:07.434+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:07.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:07.434+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:07.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:07.434+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:07.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:07.434+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:08.416 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:08.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:08.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:08.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:08.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:08.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:08.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:08.179+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:08.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:08.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:08.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:08.464+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:08.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:08.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:08.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:08.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:08.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:08.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:08.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:08.416+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:09.456 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:09.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:09.457 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:09.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:09.457 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:09.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:09.457 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:09.196+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:09.500+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:09.500+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:09.500+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:09.500+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:09.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:09.455+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:09.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:09.455+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:09.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:09.455+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:09.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:09.456+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:10.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:10.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:10.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:10.199+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:10.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:10.434+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:10.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:10.434+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:10.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:10.434+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:10.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:10.434+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:10.507+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:10.507+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:10.507+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:10.507+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:11.183+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:11.183+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:11.183+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:11.183+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:11.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:11.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:11.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:11.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:11.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:11.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:11.390+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:11.528+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:11.528+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:11.528+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:11.528+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:12.482 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:12.395+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:12.482 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:12.395+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:12.482 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:12.395+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:12.482 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:12.395+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:12.482 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:12.149+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:12.482 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:12.149+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:12.482 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:12.149+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:12.483 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:12.149+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:12.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:55:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:55:12.493+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 738 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:55:12.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:12.481+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:12.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:12.481+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:12.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:12.481+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:12.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:12.482+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:13.380 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:13.124+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:13.381 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:13.124+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:13.381 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:13.124+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:13.381 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:13.125+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:13.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:13.380+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:13.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:13.380+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:13.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:13.380+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:13.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:13.380+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:13.530+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:13.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:13.530+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:13.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:13.530+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:13.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:13.530+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:14.497 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:14.338+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:14.497 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:14.338+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:14.497 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:14.338+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:14.497 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:14.338+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:14.497 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:14.108+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:14.497 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:14.108+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:14.497 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:14.108+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:14.498 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:14.109+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:14.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:14.496+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:14.496+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:14.496+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:14.496+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:15.457 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:15.132+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:15.457 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:15.132+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:15.457 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:15.132+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:15.457 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:15.133+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:15.457 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:15.298+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:15.457 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:15.298+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:15.457 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:15.298+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:15.457 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:15.299+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:15.456+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:15.456+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:15.456+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:15.456+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:16.472 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:16.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:16.472 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:16.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:16.473 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:16.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:16.473 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:16.139+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:16.473 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:16.274+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:16.473 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:16.274+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:16.473 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:16.274+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:16.473 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:16.274+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:16.472+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:16.472+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:16.472+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:16.472+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:17.453 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:17.163+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:17.453 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:17.163+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:17.453 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:17.163+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:17.453 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:17.163+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:17.453 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:17.271+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:17.453 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:17.271+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:17.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:17.271+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:17.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:17.272+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:17.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:55:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:55:17.494+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 804 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:55:17.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:17.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:17.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:17.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:17.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:17.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:17.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:17.452+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:18.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:18.116+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:18.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:18.116+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:18.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:18.116+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:18.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:18.116+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:18.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:18.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:18.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:18.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:18.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:18.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:18.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:18.307+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:18.491+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:18.491+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:18.491+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:18.492+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:19.142+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:19.142+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:19.142+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:19.143+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:19.299+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:19.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:19.299+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:19.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:19.299+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:19.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:19.300+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:19.504+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:19.504+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:19.504+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:19.505+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:20.464 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:20.104+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:20.464 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:20.104+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:20.464 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:20.104+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:20.464 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:20.104+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:20.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:20.329+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:20.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:20.329+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:20.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:20.329+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:20.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:20.329+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:20.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:20.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:20.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:20.464+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:21.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:21.151+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:21.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:21.151+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:21.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:21.151+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:21.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:21.151+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:21.642 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:21.480+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:21.642 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:21.480+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:21.642 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:21.480+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:21.642 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:21.481+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:21.642 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:21.324+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:21.643 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:21.324+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:21.643 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:21.324+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:21.643 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:21.324+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:22.433 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:22.332+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:22.433 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:22.332+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:22.433 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:22.332+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:22.433 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:22.332+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:22.433 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:22.173+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:22.433 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:22.173+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:22.433 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:22.173+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:22.433 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:22.173+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:22.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:55:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:55:22.495+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 885 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:55:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:22.432+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:22.432+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:22.432+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:22.432+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:23.483 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:23.212+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:23.483 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:23.212+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:23.483 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:23.212+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:23.483 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:23.212+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:23.484 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:23.290+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:23.484 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:23.290+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:23.484 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:23.290+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:23.484 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:23.290+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:23.482+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:23.482+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:23.482+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:23.483+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:24.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:24.457+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:24.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:24.457+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:24.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:24.457+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:24.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:24.458+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:24.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:24.258+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:24.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:24.258+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:24.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:24.258+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:24.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:24.258+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:24.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:24.225+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:24.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:24.225+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:24.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:24.225+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:24.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:24.226+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:25.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:25.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:25.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:25.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:25.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:25.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:25.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:25.428+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:25.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:25.222+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:25.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:25.222+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:25.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:25.222+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:25.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:25.222+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:25.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:25.271+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:25.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:25.271+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:25.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:25.271+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:25.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:25.271+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:26.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:26.447+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:26.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:26.447+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:26.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:26.447+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:26.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:26.447+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:26.272+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:26.272+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:26.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:26.272+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:26.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:26.272+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:26.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:26.228+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:26.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:26.228+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:26.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:26.228+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:26.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:26.228+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:27.443 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:27.321+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:27.443 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:27.321+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:27.443 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:27.321+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:27.443 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:27.321+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:27.443 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:27.265+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:27.443 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:27.265+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:27.443 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:27.265+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:27.444 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:27.265+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:27.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:55:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:55:27.496+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 965 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:55:27.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:27.442+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:27.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:27.442+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:27.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:27.442+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:27.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:27.443+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:28.285 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:28.237+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:28.285 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:28.237+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:28.285 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:28.237+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:28.285 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:28.238+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:28.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:28.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:28.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:28.484+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:28.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:28.285+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:28.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:28.285+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:28.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:28.285+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:28.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:28.285+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:29.455 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:29.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:29.455 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:29.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:29.455 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:29.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:29.455 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:29.195+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:29.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:29.297+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:29.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:29.297+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:29.456 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:29.297+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:29.456 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:29.297+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:29.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:29.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:29.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:29.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:29.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:29.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:29.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:29.455+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:30.447 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:30.165+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:30.447 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:30.165+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:30.447 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:30.165+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:30.447 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:30.166+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:30.448 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:30.280+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:30.448 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:30.280+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:30.448 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:30.280+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:30.448 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:30.281+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:30.447+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:30.447+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:30.447+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:30.447+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:31.493 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:31.212+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:31.493 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:31.212+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:31.493 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:31.212+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:31.493 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:31.213+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:31.494 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:31.243+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:31.494 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:31.243+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:31.494 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:31.243+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:31.494 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:31.243+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:31.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:31.492+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:31.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:31.492+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:31.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:31.492+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:31.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:31.493+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:32.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:55:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:55:32.497+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 1045 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:55:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:32.250+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:32.250+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:32.250+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:32.251+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:32.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:32.252+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:32.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:32.252+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:32.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:32.252+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:32.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:32.253+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:33.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:32.513+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:33.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:32.513+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:33.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:32.513+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:33.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:32.513+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:33.530 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:33.268+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:33.530 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:33.268+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:33.530 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:33.268+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:33.530 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:33.268+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:33.530 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:33.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:33.531 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:33.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:33.531 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:33.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:33.531 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:33.486+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:33.531 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:33.270+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:33.531 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:33.270+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:33.531 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:33.270+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:33.531 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:33.270+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:34.222+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:34.222+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:34.222+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:34.223+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:34.221+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:34.221+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:34.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:34.221+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:34.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:34.221+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:34.511+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:34.511+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:34.511+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:34.511+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:35.261+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:35.261+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:35.261+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:35.262+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:35.242+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:35.242+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:35.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:35.242+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:35.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:35.243+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:35.559+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:35.559+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:35.559+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:35.560+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:36.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:36.293+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:36.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:36.293+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:36.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:36.293+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:36.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:36.293+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:36.211+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:36.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:36.211+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:36.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:36.211+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:36.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:36.212+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:36.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:36.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:36.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:36.594+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:37.499 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:37.210+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:37.499 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:37.210+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:37.499 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:37.210+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:37.499 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:37.210+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:37.499 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:37.319+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:37.499 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:37.319+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:37.499 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:37.319+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:37.499 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:37.320+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:37.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:55:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:55:37.498+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 1123 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:55:37.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:37.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:37.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:37.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:37.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:37.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:37.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:37.586+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:38.283 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:38.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:38.283 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:38.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:38.283 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:38.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:38.283 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:38.180+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:38.585 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:38.282+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:38.586 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:38.282+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:38.586 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:38.282+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:38.586 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:38.283+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:38.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:38.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:38.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:38.585+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:39.258+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:39.258+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:39.258+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:39.258+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:39.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:39.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:39.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:39.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:39.179+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:39.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:39.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:39.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:39.590+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:40.289+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:40.289+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:40.289+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:40.289+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:40.186+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:40.186+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:40.186+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:40.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:40.186+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:40.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:40.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:40.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:40.586+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:41.335+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:41.335+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:41.335+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:41.335+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:41.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:41.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:41.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:41.179+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:41.617+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:41.617+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:41.617+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:41.617+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:42.499 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:42.219+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:42.500 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:42.219+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:42.500 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:42.219+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:42.500 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:42.219+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:42.500 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:42.355+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:42.500 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:42.355+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:42.500 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:42.355+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:42.500 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:42.355+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:42.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:55:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:55:42.499+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 1204 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:55:42.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:42.646+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:42.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:42.646+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:42.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:42.646+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:42.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:42.646+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:43.390+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:43.390+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:43.390+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:43.390+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:43.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:43.190+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:43.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:43.190+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:43.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:43.190+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:43.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:43.190+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:43.620+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:43.620+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:43.620+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:43.620+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:44.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:44.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:44.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:44.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:44.151+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:44.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:44.354+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:44.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:44.354+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:44.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:44.354+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:44.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:44.354+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:44.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:44.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:44.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:44.606+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:45.160+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:45.160+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:45.160+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:45.161+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:45.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:45.328+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:45.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:45.328+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:45.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:45.328+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:45.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:45.328+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:45.607+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:45.607+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:45.607+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:45.607+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:46.156+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:46.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:46.156+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:46.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:46.156+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:46.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:46.156+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:46.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:46.286+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:46.504 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:46.286+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:46.504 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:46.286+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:46.504 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:46.286+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:46.655+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:46.655+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:46.655+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:47.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:46.656+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:47.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:47.335+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:47.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:47.335+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:47.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:47.335+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:47.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:47.335+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:47.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:47.148+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:47.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:47.148+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:47.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:47.148+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:47.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:47.148+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:47.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:55:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:55:47.500+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 1285 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:55:47.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:47.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:47.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:47.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:47.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:47.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:47.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:47.615+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:48.377 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:48.186+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:48.377 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:48.186+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:48.377 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:48.186+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:48.377 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:48.186+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:48.632 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:48.376+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:48.632 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:48.376+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:48.632 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:48.376+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:48.632 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:48.377+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:48.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:48.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:48.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:48.632+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:49.341+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:49.341+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:49.341+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:49.341+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:49.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:49.203+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:49.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:49.203+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:49.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:49.203+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:49.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:49.203+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:49.589+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:49.589+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:49.589+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:49.590+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:50.387+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:50.387+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:50.387+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:50.387+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:50.231+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:50.231+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:50.231+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:50.231+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:50.639+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:50.639+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:50.639+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:50.639+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:51.411+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:51.411+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:51.411+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:51.412+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:51.219+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:51.219+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:51.219+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:51.219+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:51.686+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:51.686+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:52.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:51.686+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:52.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:51.686+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:52.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:52.410+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:52.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:52.410+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:52.410+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:52.410+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:52.222+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:52.222+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:52.222+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:52.222+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:52.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:55:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:55:52.500+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 1370 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:55:52.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:52.655+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:52.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:52.655+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:52.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:52.655+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:52.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:52.655+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:53.455+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:53.455+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:53.455+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:53.455+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:53.226+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:53.226+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:53.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:53.226+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:53.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:53.226+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:53.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:53.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:53.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:54.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:53.665+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:54.470+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:54.470+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:54.470+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:54.470+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:54.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:54.242+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:54.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:54.242+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:54.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:54.242+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:54.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:54.242+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:54.638+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:54.638+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:54.638+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:54.638+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:55.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:55.239+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:55.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:55.239+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:55.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:55.239+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:55.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:55.239+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:56.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:55.659+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:56.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:55.659+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:56.006 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:55.659+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:56.006 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:55.659+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:56.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:55.504+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:56.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:55.504+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:56.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:55.504+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:56.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:55.505+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:56.551 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:56.264+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:56.551 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:56.264+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:56.551 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:56.264+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:56.551 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:56.264+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:56.646+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:56.646+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:56.646+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:56.646+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:57.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:56.550+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:56.550+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:56.550+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:56.551+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:57.625 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:55:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:55:57.501+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 1448 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:55:57.625 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:57.519+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:57.625 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:57.519+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:57.626 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:57.519+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:57.626 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:57.520+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:57.626 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:57.314+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:57.626 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:57.314+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:57.626 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:57.314+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:57.626 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:57.314+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:57.650+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:57.650+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:57.650+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:57.650+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:58.529 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:58.265+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:58.529 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:58.265+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:58.529 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:58.265+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:58.529 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:58.265+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:58.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:58.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:58.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:55:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:58.612+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:58.528+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:58.528+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:58.528+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:55:59.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:58.529+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:55:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:59.238+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:59.238+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:59.238+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:55:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:55:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:55:59.238+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:59.600+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:59.600+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:59.600+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:55:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:55:59.601+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:00.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:59.551+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:00.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:59.551+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:00.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:59.551+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:00.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:55:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:55:59.551+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:00.597 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:00.284+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:00.597 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:00.284+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:00.597 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:00.284+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:00.597 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:00.284+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:00.883 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:00.613+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:00.884 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:00.613+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:00.884 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:00.613+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:00.884 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:00.613+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:00.884 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:00.597+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:00.884 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:00.597+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:00.884 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:00.597+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:00.884 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:00.597+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:01.620 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:01.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:01.620 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:01.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:01.620 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:01.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:01.620 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:01.308+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:01.619+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:01.619+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:01.619+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:01.620+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:02.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:01.660+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:02.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:01.660+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:02.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:01.660+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:02.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:01.660+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:02.615 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:56:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:56:02.502+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 1534 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:56:02.615 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:02.324+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:02.615 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:02.324+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:02.616 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:02.324+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:02.616 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:02.324+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:02.621+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:02.621+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:02.621+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:02.621+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:03.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:02.614+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:03.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:02.615+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:03.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:02.615+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:03.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:02.615+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:03.618 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:03.358+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:03.618 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:03.358+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:03.618 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:03.358+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:03.618 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:03.358+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:03.955 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:03.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:03.955 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:03.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:03.956 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:03.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:03.956 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:03.618+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:03.956 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:03.661+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:03.956 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:03.661+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:03.956 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:03.662+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:03.956 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:03.662+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:04.679 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:04.629+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:04.679 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:04.629+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:04.679 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:04.629+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:04.679 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:04.629+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:04.680 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:04.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:04.680 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:04.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:04.680 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:04.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:04.680 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:04.398+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:05.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:04.678+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:05.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:04.678+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:05.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:04.678+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:05.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:04.679+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:05.640 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:05.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:05.640 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:05.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:05.640 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:05.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:05.640 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:05.389+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:06.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:05.674+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:06.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:05.674+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:06.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:05.674+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:06.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:05.674+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:05.639+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:06.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:05.639+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:06.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:05.639+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:06.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:05.640+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:06.610 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:06.348+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:06.610 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:06.348+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:06.610 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:06.348+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:06.610 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:06.348+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:07.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:06.668+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:07.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:06.668+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:07.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:06.668+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:07.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:06.669+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:06.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:06.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:06.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:07.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:06.610+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:07.631 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:56:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:56:07.503+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 1627 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:56:07.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:07.370+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:07.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:07.370+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:07.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:07.370+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:07.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:07.370+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:08.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:07.687+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:08.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:07.687+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:08.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:07.687+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:08.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:07.687+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:07.653+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:07.653+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:07.653+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:08.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:07.653+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:08.606 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:08.330+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:08.606 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:08.330+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:08.607 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:08.330+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:08.607 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:08.330+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:08.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:08.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:08.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:08.606+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:09.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:08.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:09.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:08.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:09.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:08.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:09.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:08.679+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:09.654 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:09.318+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:09.655 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:09.318+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:09.655 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:09.318+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:09.655 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:09.318+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:10.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:09.704+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:10.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:09.704+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:10.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:09.704+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:10.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:09.704+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:09.653+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:09.653+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:09.653+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:09.653+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:10.668 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:10.277+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:10.668 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:10.277+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:10.668 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:10.277+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:10.668 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:10.277+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:10.688+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:10.688+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:10.688+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:10.689+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:11.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:10.666+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:11.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:10.666+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:11.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:10.666+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:11.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:10.667+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:11.638 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:11.270+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:11.638 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:11.270+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:11.638 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:11.270+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:11.639 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:11.270+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:11.701+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:11.701+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:11.701+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:11.701+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:12.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:11.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:11.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:11.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:11.637+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:12.242+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:12.242+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:12.242+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:12.243+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:13.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:56:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:56:12.503+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 1723 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:56:13.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:12.685+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:13.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:12.685+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:13.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:12.685+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:13.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:12.685+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:13.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:12.685+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:13.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:12.685+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:13.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:12.685+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:13.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:12.685+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:13.226+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:13.227+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:13.227+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:13.227+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:14.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:13.702+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:14.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:13.702+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:14.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:13.702+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:14.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:13.702+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:14.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:13.711+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:14.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:13.711+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:14.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:13.711+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:14.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:13.711+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:14.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:14.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:14.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:14.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:14.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:14.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:14.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:14.178+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:15.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:14.698+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:15.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:14.698+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:15.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:14.698+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:15.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:14.698+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:15.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:14.750+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:15.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:14.750+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:15.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:14.750+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:15.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:14.750+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:15.190+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:15.191+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:15.191+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:15.191+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:16.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:15.702+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:16.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:15.702+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:16.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:15.702+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:16.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:15.702+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:15.797+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:15.797+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:16.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:15.797+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:16.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:15.797+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:16.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:16.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:16.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:16.209+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:16.700+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:16.700+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:16.700+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:16.700+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:16.831+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:16.831+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:16.831+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:17.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:16.831+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:17.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:17.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:17.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:17.210+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:17.867 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:56:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:56:17.504+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 1817 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:56:17.867 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:17.719+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:17.868 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:17.719+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:17.868 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:17.719+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:17.868 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:17.719+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:18.198 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:17.865+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:18.198 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:17.866+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:18.199 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:17.866+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:18.199 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:17.866+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:18.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:18.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:18.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:18.197+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:18.826+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:18.826+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:18.826+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:18.827+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:19.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:18.732+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:19.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:18.732+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:19.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:18.732+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:19.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:18.732+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:19.234+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:19.234+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:19.234+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:19.234+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:19.742+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:19.742+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:19.742+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:19.742+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:19.795+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:19.795+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:19.795+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:19.795+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:20.237+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:20.237+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:20.237+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:20.237+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:21.191 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:20.754+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:21.191 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:20.754+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:21.191 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:20.754+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:21.192 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:20.755+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:21.192 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:20.806+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:21.192 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:20.806+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:21.192 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:20.806+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:21.192 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:20.806+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:21.190+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:21.190+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:21.190+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:21.190+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:21.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:21.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:21.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:21.802+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:21.848+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:21.848+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:21.848+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:21.848+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:22.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:22.183+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:22.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:22.183+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:22.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:22.183+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:22.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:22.183+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:22.766 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:56:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:56:22.505+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 1913 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:56:23.179 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:22.820+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:23.180 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:22.820+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:23.180 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:22.820+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:23.180 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:22.820+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:23.180 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:22.765+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:23.180 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:22.765+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:23.181 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:22.765+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:23.181 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:22.765+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:23.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:23.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:23.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:23.178+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:23.828 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:23.750+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:23.828 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:23.750+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:23.829 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:23.750+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:23.829 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:23.750+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:24.157 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:23.828+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:24.158 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:23.828+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:24.158 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:23.828+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:24.158 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:23.828+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:24.157+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:24.157+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:24.157+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:24.157+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:25.130 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:24.855+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:25.130 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:24.855+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:25.130 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:24.855+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:25.130 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:24.855+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:25.130 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:24.784+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:25.131 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:24.784+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:25.131 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:24.784+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:25.131 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:24.784+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:25.459 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:25.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:25.460 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:25.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:25.460 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:25.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:25.460 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:25.129+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:25.973 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T20:56:25.972+0000 7fbf2bab8700 0 monclient(hunting): authenticate timed out after 300 2023-12-16T20:56:25.973 INFO:teuthology.orchestra.run.smithi052.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-16T20:56:26.160 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:25.888+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:26.161 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:25.888+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:26.161 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:25.888+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:26.161 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:25.888+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:26.161 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:25.776+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:26.161 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:25.776+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:26.161 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:25.776+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:26.161 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:25.776+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:26.432 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:26.159+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:26.432 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:26.159+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:26.432 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:26.159+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:26.432 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:26.160+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:26.460 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2023-12-16T20:56:26.735 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T20:56:27.125 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:26.775+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:27.125 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:26.775+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:27.125 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:26.775+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:27.125 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:26.775+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:27.125 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:26.877+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:27.126 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:26.877+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:27.126 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:26.877+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:27.126 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:26.878+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:27.472 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:27.202+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:27.473 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:27.202+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:27.473 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:27.202+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:27.473 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:27.202+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:27.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:56:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:56:27.506+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 2007 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:56:28.241 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:27.783+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:28.242 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:27.783+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:28.242 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:27.783+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:28.242 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:27.784+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:28.242 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:27.927+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:28.242 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:27.927+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:28.242 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:27.927+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:28.242 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:27.927+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:28.241+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:28.241+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:28.241+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:28.241+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:28.917 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:28.774+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:28.917 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:28.774+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:28.917 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:28.774+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:28.917 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:28.774+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:29.206 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:28.916+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:29.206 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:28.916+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:29.206 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:28.916+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:29.206 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:28.917+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:29.205+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:29.205+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:29.205+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:29.206+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:30.203 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:29.783+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:30.203 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:29.783+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:30.203 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:29.783+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:30.203 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:29.784+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:30.203 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:29.959+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:30.203 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:29.959+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:30.203 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:29.959+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:30.203 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:29.959+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:30.202+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:30.202+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:30.202+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:30.202+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:30.942+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:30.942+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:30.942+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:30.943+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:30.751+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:30.751+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:30.751+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:30.751+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:31.247+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:31.247+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:31.247+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:31.247+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:31.909 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:31.777+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:31.909 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:31.777+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:31.909 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:31.777+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:31.909 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:31.777+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:32.221 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:31.908+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:32.222 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:31.909+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:32.222 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:31.909+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:32.222 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:31.909+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:32.221+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:32.221+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:32.221+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:32.221+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:32.799 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:56:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:56:32.507+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 2101 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:56:33.216 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:32.798+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:33.216 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:32.798+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:33.216 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:32.798+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:33.216 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:32.799+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:33.217 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:32.928+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:33.217 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:32.928+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:33.217 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:32.928+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:33.217 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:32.929+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:33.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:33.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:33.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:33.215+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:33.814+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:33.814+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:33.814+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:33.814+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:34.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:33.921+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:34.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:33.921+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:34.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:33.921+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:34.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:33.921+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:34.676 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:34.259+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:34.676 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:34.259+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:34.676 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:34.259+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:34.676 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:34.259+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:34.941 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:34.809+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:34.941 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:34.809+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:34.941 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:34.809+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:34.941 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:34.809+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:35.220 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:34.940+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:35.221 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:34.940+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:35.221 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:34.940+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:35.221 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:34.941+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:35.220+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:35.220+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:35.220+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:35.220+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:36.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:35.856+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:36.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:35.856+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:36.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:35.856+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:36.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:35.856+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:36.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:35.985+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:36.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:35.985+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:36.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:35.985+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:36.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:35.986+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:36.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:36.253+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:36.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:36.253+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:36.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:36.253+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:36.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:36.253+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:37.204 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:37.012+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:37.205 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:37.012+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:37.205 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:37.012+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:37.205 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:37.013+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:37.205 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:36.863+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:37.205 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:36.863+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:37.205 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:36.863+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:37.205 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:36.863+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:37.206 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:37.204+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:37.206 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:37.204+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:37.206 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:37.204+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:37.204+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:37.876 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:56:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:56:37.508+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 2195 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:56:38.159 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:37.876+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:38.159 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:37.876+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:38.159 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:37.876+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:38.159 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:37.876+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:38.159 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:38.014+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:38.159 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:38.014+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:38.160 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:38.014+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:38.160 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:38.015+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:38.433 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:38.158+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:38.434 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:38.158+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:38.434 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:38.158+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:38.434 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:38.158+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:39.208 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:38.916+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:39.208 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:38.916+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:39.208 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:38.916+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:39.208 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:38.916+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:39.208 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:39.017+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:39.208 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:39.017+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:39.208 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:39.017+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:39.209 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:39.017+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:39.207+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:39.207+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:39.207+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:39.207+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:40.203 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:39.931+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:40.203 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:39.931+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:40.203 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:39.931+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:40.203 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:39.931+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:40.204 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:39.995+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:40.204 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:39.995+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:40.204 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:39.995+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:40.204 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:39.995+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:40.202+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:40.202+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:40.202+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:40.203+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:41.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:40.931+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:41.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:40.931+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:41.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:40.931+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:41.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:40.932+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:41.175 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:40.950+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:41.175 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:40.950+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:41.175 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:40.950+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:41.175 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:40.951+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:41.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:41.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:41.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:41.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:41.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:41.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:41.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:41.174+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:42.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:41.956+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:42.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:41.956+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:42.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:41.956+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:42.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:41.956+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:42.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:41.961+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:42.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:41.961+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:42.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:41.961+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:42.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:41.961+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:42.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:42.155+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:42.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:42.155+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:42.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:42.155+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:42.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:42.155+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:42.910 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:56:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:56:42.509+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 2291 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:56:43.190 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:42.912+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:43.190 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:42.912+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:43.190 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:42.912+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:43.190 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:42.913+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:43.191 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:42.909+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:43.191 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:42.909+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:43.191 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:42.909+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:43.191 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:42.910+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:43.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:43.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:43.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:43.190+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:44.172 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:43.870+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:44.172 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:43.870+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:44.172 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:43.870+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:44.172 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:43.871+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:44.173 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:43.958+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:44.173 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:43.958+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:44.173 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:43.958+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:44.173 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:43.959+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:44.171+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:44.171+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:44.171+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:44.171+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:45.196 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:44.920+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:45.196 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:44.920+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:45.196 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:44.920+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:45.196 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:44.921+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:45.196 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:44.914+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:45.196 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:44.914+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:45.196 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:44.914+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:45.196 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:44.914+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:45.195+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:45.195+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:45.195+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:45.195+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:46.182 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:45.906+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:46.182 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:45.906+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:46.182 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:45.906+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:46.182 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:45.906+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:46.182 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:45.953+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:46.183 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:45.953+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:46.183 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:45.953+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:46.183 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:45.953+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:46.182+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:46.182+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:46.182+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:46.182+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:47.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:46.938+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:47.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:46.938+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:47.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:46.938+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:47.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:46.939+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:47.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:46.954+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:47.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:46.954+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:47.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:46.954+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:47.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:46.954+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:47.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:47.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:47.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:47.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:47.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:47.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:47.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:47.151+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:47.901 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:56:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:56:47.510+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 2385 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:56:48.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:47.900+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:48.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:47.900+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:48.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:47.900+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:48.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:47.901+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:48.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:48.121+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:48.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:48.121+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:48.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:48.121+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:48.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:48.121+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:48.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:47.977+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:48.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:47.977+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:48.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:47.977+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:48.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:47.978+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:49.169 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:48.966+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:49.169 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:48.966+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:49.169 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:48.966+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:49.170 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:48.967+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:49.170 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:48.911+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:49.170 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:48.911+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:49.170 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:48.911+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:49.170 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:48.911+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:49.170 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:49.169+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:49.170 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:49.169+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:49.171 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:49.169+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:49.171 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:49.169+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:50.213 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:49.992+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:50.213 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:49.992+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:50.213 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:49.992+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:50.213 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:49.993+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:50.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:49.912+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:50.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:49.912+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:50.214 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:49.912+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:50.214 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:49.912+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:50.212+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:50.212+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:50.212+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:50.212+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:51.169 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:50.998+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:51.169 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:50.998+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:51.170 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:50.998+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:51.170 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:50.999+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:51.170 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:50.891+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:51.170 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:50.891+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:51.170 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:50.891+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:51.170 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:50.891+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:51.169+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:51.169+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:51.169+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:51.169+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:52.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:51.950+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:52.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:51.950+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:52.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:51.950+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:52.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:51.951+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:52.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:52.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:52.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:52.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:52.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:52.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:52.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:52.131+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:52.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:51.934+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:52.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:51.934+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:52.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:51.934+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:52.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:51.934+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:52.912 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:56:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:56:52.511+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 2480 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:56:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:52.955+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:52.955+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:52.955+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:52.955+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:53.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:52.912+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:53.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:52.912+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:53.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:52.912+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:53.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:52.912+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:53.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:53.114+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:53.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:53.114+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:53.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:53.114+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:53.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:53.115+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:54.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:53.886+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:54.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:53.886+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:54.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:53.886+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:54.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:53.887+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:54.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:54.124+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:54.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:54.124+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:54.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:54.124+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:54.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:54.124+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:54.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:53.967+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:54.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:53.967+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:54.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:53.967+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:54.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:53.967+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:55.168 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:55.005+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:55.168 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:55.005+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:55.168 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:55.005+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:55.168 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:55.005+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:55.168 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:54.906+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:55.168 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:54.906+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:55.168 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:54.906+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:55.168 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:54.906+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:55.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:55.167+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:55.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:55.167+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:55.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:55.167+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:55.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:55.167+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:56.180 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:55.969+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:56.180 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:55.969+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:56.180 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:55.969+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:56.181 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:55.969+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:56.181 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:55.898+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:56.181 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:55.898+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:56.181 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:55.898+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:56.181 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:55.898+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:56.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:56.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:56.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:56.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:56.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:56.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:56.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:56.180+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:57.160 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:57.000+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:57.160 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:57.000+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:57.160 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:57.000+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:57.160 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:57.001+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:57.160 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:56.873+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:57.161 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:56.873+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:57.161 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:56.873+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:57.161 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:56.873+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:57.159+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:57.159+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:57.159+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:57.160+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:57.837 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:56:57.512+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 2575 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:56:58.147 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:57.966+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:58.147 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:57.966+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:58.147 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:57.966+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:58.147 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:57.966+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:58.147 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:57.836+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:58.147 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:57.836+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:58.147 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:57.836+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:58.148 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:57.837+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:58.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:58.146+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:58.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:58.146+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:58.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:58.146+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:58.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:58.146+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:59.151 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:58.992+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:59.152 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:58.992+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:59.152 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:58.992+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:56:59.152 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:56:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:56:58.992+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:59.152 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:58.877+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:59.152 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:58.877+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:59.152 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:58.877+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:56:59.152 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:58.878+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:56:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:59.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:59.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:59.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:56:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:56:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:56:59.151+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:00.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:00.034+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:00.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:00.035+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:00.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:00.035+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:00.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:00.035+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:00.174 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:59.866+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:00.174 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:59.866+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:00.174 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:59.866+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:00.174 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:56:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:56:59.866+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:00.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:00.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:00.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:00.173+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:01.128 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:01.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:01.129 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:01.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:01.129 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:01.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:01.129 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:01.082+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:01.129 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:00.876+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:01.129 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:00.876+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:01.129 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:00.876+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:01.129 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:00.876+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:01.128+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:01.128+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:01.128+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:01.128+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:01.869+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:01.869+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:01.869+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:01.869+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:02.128+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:02.128+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:02.128+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:02.129+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:02.150+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:02.150+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:02.150+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:02.150+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:02.860 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:57:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:57:02.513+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 2670 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:57:03.138 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:02.860+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:03.138 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:02.860+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:03.138 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:02.860+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:03.139 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:02.861+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:03.139 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:03.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:03.139 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:03.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:03.139 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:03.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:03.139 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:03.082+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:03.138+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:03.138+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:03.138+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:03.138+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:04.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:04.095+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:04.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:04.095+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:04.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:04.095+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:04.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:04.096+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:04.092+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:04.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:04.092+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:04.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:04.092+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:04.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:04.093+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:03.850+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:03.850+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:03.850+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:03.851+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:04.832+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:04.832+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:04.832+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:04.832+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:05.069+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:05.069+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:05.069+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:05.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:05.070+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:05.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:05.055+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:05.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:05.055+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:05.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:05.055+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:05.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:05.055+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:06.088 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:06.036+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:06.088 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:06.036+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:06.088 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:06.036+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:06.088 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:06.037+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:06.088 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:05.824+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:06.088 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:05.824+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:06.088 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:05.824+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:06.088 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:05.824+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:06.087+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:06.087+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:06.087+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:06.087+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:07.109 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:06.786+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:07.109 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:06.786+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:07.109 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:06.786+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:07.109 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:06.786+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:07.109 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:06.994+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:07.109 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:06.994+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:07.109 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:06.994+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:07.110 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:06.995+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:07.108+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:07.108+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:07.108+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:07.108+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:08.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:57:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:57:07.514+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 2763 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:57:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:07.760+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:07.760+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:07.760+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:07.760+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:08.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:08.024+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:08.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:08.024+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:08.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:08.024+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:08.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:08.024+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:08.075+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:08.075+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:08.075+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:08.075+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:08.993 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:08.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:08.993 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:08.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:08.993 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:08.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:08.993 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:08.727+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:09.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:08.993+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:09.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:08.993+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:09.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:08.993+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:09.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:08.993+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:09.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:09.045+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:09.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:09.045+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:09.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:09.045+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:09.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:09.045+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:10.046 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:09.999+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:10.046 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:09.999+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:10.046 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:09.999+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:10.046 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:09.999+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:10.046 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:09.763+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:10.046 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:09.763+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:10.047 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:09.763+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:10.047 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:09.763+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:10.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:10.046+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:10.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:10.046+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:10.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:10.046+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:10.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:10.046+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:11.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:10.806+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:11.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:10.806+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:11.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:10.806+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:11.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:10.806+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:11.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:11.029+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:11.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:11.029+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:11.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:11.029+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:11.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:11.029+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:11.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:11.001+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:11.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:11.001+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:11.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:11.001+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:11.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:11.001+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:12.000 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:11.818+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:12.000 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:11.818+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:12.000 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:11.818+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:12.000 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:11.818+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:12.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:12.000+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:12.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:12.000+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:12.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:12.000+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:12.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:12.000+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:12.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:12.029+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:12.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:12.029+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:12.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:12.029+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:12.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:12.029+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:12.847 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:57:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:57:12.515+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 2857 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:57:13.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:13.026+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:13.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:13.026+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:13.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:13.026+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:13.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:13.027+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:13.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:13.005+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:13.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:13.005+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:13.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:13.005+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:13.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:13.005+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:13.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:12.847+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:13.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:12.847+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:13.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:12.847+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:13.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:12.847+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:14.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:14.008+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:14.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:14.008+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:14.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:14.008+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:14.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:14.009+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:14.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:13.826+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:14.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:13.826+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:14.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:13.827+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:14.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:13.827+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:14.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:14.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:14.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:14.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:14.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:14.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:14.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:14.039+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:14.832+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:14.832+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:14.832+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:14.832+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:15.035+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:15.035+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:15.035+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:15.036+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:15.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:15.058+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:15.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:15.058+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:15.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:15.058+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:15.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:15.058+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:16.024+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:16.024+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:16.024+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:16.025+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:16.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:16.018+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:16.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:16.018+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:16.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:16.018+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:16.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:16.018+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:16.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:15.828+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:16.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:15.828+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:16.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:15.828+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:16.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:15.828+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:17.056 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:16.787+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:17.056 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:16.787+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:17.056 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:16.787+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:17.056 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:16.787+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:17.056 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:17.027+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:17.056 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:17.027+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:17.057 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:17.027+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:17.057 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:17.027+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:17.054+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:17.054+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:17.054+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:17.055+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:18.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:57:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:57:17.516+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 2951 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:57:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:17.759+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:17.759+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:17.759+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:17.759+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:18.042+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:18.042+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:18.042+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:18.042+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:18.051+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:18.051+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:18.051+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:18.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:18.051+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:19.098 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:18.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:19.098 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:18.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:19.098 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:18.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:19.099 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:18.797+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:19.099 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:19.020+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:19.099 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:19.020+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:19.099 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:19.020+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:19.099 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:19.020+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:19.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:19.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:19.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:19.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:19.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:19.098+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:20.128 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:19.773+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:20.128 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:19.773+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:20.128 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:19.773+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:20.128 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:19.773+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:20.129 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:19.982+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:20.129 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:19.982+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:20.129 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:19.982+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:20.129 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:19.982+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:20.128+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:20.128+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:20.128+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:20.128+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:21.146 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:20.953+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:21.146 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:20.953+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:21.146 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:20.953+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:21.146 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:20.953+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:21.146 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:20.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:21.146 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:20.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:21.146 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:20.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:21.147 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:20.761+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:21.146+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:21.146+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:21.146+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:21.146+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:21.992 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:21.788+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:21.992 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:21.788+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:21.993 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:21.788+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:21.993 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:21.788+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:22.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:21.992+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:22.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:21.992+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:22.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:21.992+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:22.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:21.992+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:22.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:22.194+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:22.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:22.194+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:22.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:22.194+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:22.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:22.195+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:22.798 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:57:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:57:22.517+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 3046 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:57:23.166 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:23.021+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:23.166 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:23.021+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:23.166 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:23.021+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:23.166 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:23.021+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:23.166 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:22.798+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:23.166 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:22.798+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:23.167 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:22.798+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:23.167 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:22.798+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:23.166+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:23.166+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:23.166+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:23.166+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:24.129 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:23.989+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:24.129 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:23.989+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:24.129 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:23.989+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:24.129 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:23.990+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:24.130 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:23.797+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:24.130 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:23.797+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:24.130 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:23.797+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:24.130 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:23.797+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:24.128+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:24.128+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:24.128+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:24.128+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:25.178 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:24.837+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:25.178 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:24.837+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:25.178 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:24.837+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:25.178 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:24.837+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:25.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:25.028+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:25.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:25.028+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:25.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:25.028+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:25.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:25.028+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:25.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:25.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:25.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:25.178+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:26.174 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:25.883+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:26.174 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:25.883+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:26.174 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:25.883+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:26.174 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:25.883+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:26.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:26.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:26.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:26.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:26.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:26.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:26.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:26.039+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:26.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:26.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:26.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:26.173+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:27.181 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:26.915+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:27.181 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:26.915+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:27.181 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:26.916+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:27.181 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:26.916+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:27.181 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:27.053+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:27.181 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:27.053+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:27.181 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:27.053+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:27.181 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:27.053+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:27.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:27.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:27.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:27.180+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:27.912 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:57:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:57:27.519+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 3139 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:57:28.206 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:27.912+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:28.206 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:27.912+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:28.206 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:27.912+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:28.207 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:27.912+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:28.207 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:28.075+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:28.207 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:28.075+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:28.207 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:28.075+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:28.207 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:28.076+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:28.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:28.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:28.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:28.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:28.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:28.206+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:29.201 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:29.099+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:29.203 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:29.099+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:29.203 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:29.099+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:29.203 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:29.099+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:29.203 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:28.877+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:29.203 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:28.877+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:29.203 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:28.877+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:29.203 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:28.877+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:29.198+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:29.198+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:29.198+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:29.198+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:30.176 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:29.909+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:30.176 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:29.909+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:30.176 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:29.909+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:30.176 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:29.909+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:30.176 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:30.126+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:30.176 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:30.126+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:30.177 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:30.126+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:30.177 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:30.127+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:30.176+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:30.176+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:30.176+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:30.176+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:31.144 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:30.864+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:31.144 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:30.864+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:31.145 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:30.864+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:31.145 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:30.864+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:31.162+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:31.162+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:31.162+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:31.163+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:31.145+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:31.145+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:31.145+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:31.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:31.145+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:32.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:31.835+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:31.835+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:31.835+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:31.835+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:32.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:32.200+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:32.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:32.200+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:32.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:32.200+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:32.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:32.200+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:32.165+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:32.165+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:32.165+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:32.165+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:32.855 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:57:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:57:32.520+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 3233 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:57:33.173 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:32.855+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:33.173 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:32.855+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:33.173 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:32.855+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:33.173 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:32.855+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:33.171+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:33.171+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:33.171+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:33.171+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:33.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:33.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:33.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:33.180+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:34.143 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:33.831+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:34.143 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:33.831+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:34.144 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:33.831+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:34.144 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:33.832+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:34.142+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:34.142+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:34.142+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:34.142+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:34.188+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:34.188+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:34.188+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:34.188+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:35.198 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:34.862+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:35.199 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:34.862+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:35.199 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:34.862+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:35.199 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:34.862+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:35.199 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:35.102+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:35.199 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:35.102+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:35.199 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:35.102+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:35.199 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:35.102+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:35.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:35.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:35.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:35.197+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:36.115 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:36.064+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:36.116 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:36.064+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:36.116 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:36.064+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:36.116 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:36.064+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:36.116 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:35.833+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:36.116 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:35.833+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:36.116 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:35.833+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:36.116 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:35.833+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:36.216+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:36.216+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:36.216+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:36.216+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:37.110 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:36.819+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:37.110 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:36.819+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:37.111 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:36.819+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:37.111 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:36.819+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:37.227+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:37.227+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:37.227+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:37.228+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:37.110+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:37.110+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:37.110+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:37.110+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:37.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:57:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:57:37.520+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 3327 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:57:38.118 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:37.841+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:38.118 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:37.841+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:38.118 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:37.841+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:38.118 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:37.842+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:38.117+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:38.117+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:38.117+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:38.117+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:38.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:38.181+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:38.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:38.181+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:38.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:38.181+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:38.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:38.181+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:39.153 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:38.797+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:39.153 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:38.797+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:39.153 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:38.797+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:39.154 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:38.797+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:39.152+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:39.152+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:39.152+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:39.152+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:39.172+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:39.172+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:39.172+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:39.172+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:40.138 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:39.823+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:40.138 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:39.823+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:40.138 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:39.823+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:40.138 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:39.823+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:40.137+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:40.137+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:40.137+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:40.137+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:40.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:40.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:40.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:40.215+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:41.169 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:40.800+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:41.169 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:40.800+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:41.169 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:40.800+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:41.169 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:40.800+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:41.168+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:41.168+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:41.168+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:41.168+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:41.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:41.225+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:41.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:41.225+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:41.225+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:41.225+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:41.798+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:41.798+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:41.798+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:41.798+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:42.128+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:42.128+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:42.128+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:42.128+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:42.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:42.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:42.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:42.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:42.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:42.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:42.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:42.189+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:42.791 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:57:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:57:42.521+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 3422 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:57:43.178 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:42.789+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:43.179 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:42.789+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:43.179 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:42.789+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:43.179 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:42.790+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:43.218+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:43.218+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:43.218+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:43.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:43.218+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:43.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:43.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:43.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:43.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:43.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:43.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:43.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:43.178+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:43.764+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:43.764+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:43.764+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:43.764+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:44.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:44.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:44.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:44.209+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:44.199+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:44.199+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:44.199+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:44.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:44.199+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:45.195 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:44.809+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:45.195 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:44.809+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:45.195 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:44.809+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:45.195 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:44.809+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:45.194+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:45.194+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:45.194+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:45.194+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:45.241+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:45.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:45.241+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:45.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:45.241+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:45.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:45.241+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:46.179 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:45.781+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:46.179 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:45.781+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:46.179 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:45.781+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:46.179 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:45.781+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:46.266+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:46.266+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:46.266+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:46.267+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:46.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:46.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:46.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:46.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:46.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:46.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:46.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:46.178+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:46.738+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:46.738+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:46.738+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:46.738+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:47.286+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:47.286+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:47.286+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:47.286+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:47.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:47.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:47.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:47.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:47.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:47.192+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:48.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:57:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:57:47.522+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 3516 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:57:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:47.741+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:47.741+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:47.741+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:47.741+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:48.260 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:48.144+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:48.260 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:48.144+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:48.260 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:48.144+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:48.260 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:48.144+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:48.746 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:48.260+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:48.746 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:48.260+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:48.746 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:48.260+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:48.746 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:48.260+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:48.745+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:48.745+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:48.745+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:48.745+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:49.298+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:49.298+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:49.298+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:49.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:49.298+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:49.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:49.128+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:49.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:49.128+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:49.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:49.128+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:49.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:49.128+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:49.740+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:49.740+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:49.740+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:49.740+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:50.319+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:50.319+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:50.319+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:50.319+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:50.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:50.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:50.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:50.098+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:50.702+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:50.702+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:50.702+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:51.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:50.703+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:51.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:51.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:51.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:51.338+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:51.110+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:51.110+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:51.110+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:51.110+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:51.721+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:51.721+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:51.721+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:51.721+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:52.373+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:52.373+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:52.373+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:52.374+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:52.135+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:52.135+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:52.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:52.135+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:52.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:52.135+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:53.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:57:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:57:52.524+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 3612 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:57:53.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:52.733+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:53.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:52.733+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:53.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:52.733+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:53.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:52.734+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:53.374+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:53.374+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:53.374+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:53.374+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:53.172+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:53.172+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:53.172+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:53.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:53.173+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:53.726+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:53.726+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:53.726+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:53.726+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:54.405+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:54.405+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:54.405+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:54.405+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:54.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:54.199+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:54.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:54.199+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:54.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:54.199+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:54.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:54.199+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:54.745+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:54.745+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:54.745+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:54.745+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:55.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:55.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:55.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:55.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:55.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:55.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:55.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:55.198+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:55.717 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:55.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:55.717 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:55.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:55.717 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:55.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:55.718 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:55.454+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:55.716+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:55.716+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:55.716+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:55.717+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:56.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:56.422+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:56.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:56.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:56.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:56.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:56.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:56.423+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:56.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:56.200+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:56.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:56.201+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:56.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:56.201+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:56.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:56.201+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:56.688+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:56.688+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:56.688+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:56.688+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:57.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:57.377+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:57.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:57.377+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:57.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:57.377+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:57.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:57.378+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:57.152+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:57.152+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:57.152+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:57.152+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:58.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:57:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:57:57.525+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 3708 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:57:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:57.728+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:57.728+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:57.728+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:57.728+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:58.356 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:58.136+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:58.356 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:58.136+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:58.357 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:58.136+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:58.357 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:58.137+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:58.742 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:58.356+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:58.743 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:58.356+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:58.743 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:58.356+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:58.743 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:58.356+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:59.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:58.742+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:58.742+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:58.742+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:57:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:58.742+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:59.349+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:59.349+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:59.350+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:57:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:57:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:57:59.350+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:57:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:59.154+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:59.154+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:59.154+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:57:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:57:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:57:59.154+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:00.121 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:59.786+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:00.121 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:59.786+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:00.121 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:59.786+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:00.121 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:57:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:57:59.786+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:00.331+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:00.331+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:00.331+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:00.331+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:00.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:00.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:00.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:00.121+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:01.096 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:00.777+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:01.096 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:00.777+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:01.096 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:00.777+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:01.096 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:00.778+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:01.324+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:01.324+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:01.324+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:01.325+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:01.095+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:01.095+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:01.095+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:01.096+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:01.738+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:01.738+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:01.738+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:01.738+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:02.305+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:02.306+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:02.306+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:02.306+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:02.100+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:02.100+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:02.100+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:02.100+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:03.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:58:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:58:02.526+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 3806 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:58:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:02.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:02.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:02.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:02.728+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:03.282+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:03.282+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:03.282+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:03.283+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:03.083+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:03.083+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:03.083+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:03.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:03.083+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:03.716+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:03.716+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:03.716+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:03.716+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:04.320+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:04.320+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:04.320+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:04.321+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:04.080+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:04.080+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:04.080+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:04.080+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:04.692+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:04.692+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:04.692+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:04.693+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:05.344 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:05.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:05.344 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:05.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:05.344 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:05.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:05.344 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:05.075+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:05.738 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:05.344+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:05.738 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:05.344+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:05.738 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:05.344+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:05.738 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:05.344+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:05.737+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:05.737+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:05.737+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:05.738+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:06.300+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:06.300+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:06.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:06.300+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:06.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:06.301+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:06.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:06.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:06.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:06.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:06.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:06.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:06.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:06.099+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:06.740+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:06.740+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:06.740+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:06.740+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:07.273+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:07.273+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:07.273+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:07.274+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:07.064+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:07.064+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:07.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:07.064+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:07.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:07.064+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:08.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:58:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:58:07.527+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 3900 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:58:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:07.770+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:07.770+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:07.770+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:07.771+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:08.282 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:08.091+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:08.282 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:08.091+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:08.282 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:08.091+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:08.283 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:08.091+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:08.281+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:08.282+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:08.282+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:08.282+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:09.051 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:08.799+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:09.051 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:08.799+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:09.051 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:08.799+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:09.051 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:08.800+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:09.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:09.270+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:09.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:09.270+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:09.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:09.270+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:09.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:09.270+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:09.051+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:09.051+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:09.051+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:09.051+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:10.088 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:09.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:10.088 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:09.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:10.089 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:09.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:10.089 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:09.752+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:10.309+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:10.309+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:10.309+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:10.309+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:10.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:10.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:10.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:10.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:10.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:10.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:10.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:10.088+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:11.098 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:10.759+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:11.098 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:10.759+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:11.098 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:10.759+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:11.098 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:10.760+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:11.359 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:11.097+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:11.359 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:11.097+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:11.359 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:11.097+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:11.359 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:11.097+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:11.652 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:11.358+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:11.653 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:11.358+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:11.653 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:11.358+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:11.653 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:11.359+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:11.779+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:11.779+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:11.779+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:11.779+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:12.333+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:12.333+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:12.333+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:12.333+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:12.104+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:12.104+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:12.104+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:12.104+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:12.821 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:58:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:58:12.528+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 3995 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:58:13.076 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:12.820+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:13.076 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:12.820+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:13.076 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:12.820+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:13.077 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:12.821+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:13.293+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:13.293+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:13.293+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:13.293+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:13.076+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:13.076+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:13.076+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:13.076+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:13.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:13.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:13.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:13.777+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:14.308+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:14.308+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:14.308+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:14.309+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:14.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:14.114+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:14.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:14.114+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:14.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:14.114+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:14.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:14.115+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:15.088 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:14.773+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:15.088 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:14.773+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:15.088 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:14.773+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:15.088 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:14.774+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:15.282+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:15.282+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:15.282+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:15.283+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:15.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:15.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:15.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:15.088+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:16.047 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:15.766+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:16.047 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:15.766+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:16.047 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:15.766+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:16.047 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:15.767+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:16.046+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:16.046+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:16.046+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:16.047+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:16.265+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:16.265+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:16.265+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:16.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:16.266+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:17.018 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:16.754+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:17.018 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:16.754+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:17.018 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:16.754+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:17.018 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:16.754+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:17.307 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:17.017+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:17.307 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:17.017+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:17.307 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:17.017+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:17.307 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:17.018+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:17.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:58:17.529+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 4089 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:58:17.654 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:17.306+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:17.654 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:17.306+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:17.654 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:17.306+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:17.654 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:17.307+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:17.977 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:17.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:17.977 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:17.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:17.977 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:17.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:17.977 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:17.727+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:18.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:17.977+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:18.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:17.977+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:18.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:17.977+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:18.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:17.977+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:18.710 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:18.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:18.710 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:18.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:18.710 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:18.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:18.710 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:18.338+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:18.970 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:18.709+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:18.970 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:18.709+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:18.970 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:18.709+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:18.970 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:18.710+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:19.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:18.970+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:19.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:18.970+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:19.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:18.970+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:19.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:18.970+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:19.702 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:19.294+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:19.702 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:19.294+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:19.702 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:19.294+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:19.702 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:19.295+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:20.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:19.950+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:20.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:19.950+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:20.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:19.950+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:20.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:19.950+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:19.701+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:19.701+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:19.701+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:19.702+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:20.249+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:20.249+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:20.249+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:20.249+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:20.747+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:20.747+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:20.747+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:20.748+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:20.986+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:20.986+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:20.986+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:20.986+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:21.238+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:21.238+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:21.238+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:21.239+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:21.749+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:21.749+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:21.749+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:21.750+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:22.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:22.030+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:22.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:22.030+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:22.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:22.030+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:22.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:22.031+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:22.205+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:22.205+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:22.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:22.205+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:22.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:22.205+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:23.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:58:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:58:22.530+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 4185 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:58:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:22.768+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:22.768+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:22.768+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:22.769+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:23.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:23.213+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:23.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:23.213+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:23.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:23.213+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:23.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:23.213+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:23.077+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:23.077+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:23.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:23.077+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:23.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:23.077+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:23.750+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:23.750+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:23.750+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:23.750+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:24.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:24.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:24.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:24.236+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:24.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:24.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:24.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:24.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:24.074+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:24.703+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:24.703+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:24.703+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:24.703+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:25.246+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:25.246+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:25.246+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:25.247+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:25.053+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:25.053+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:25.053+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:25.053+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:25.698+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:25.698+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:25.698+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:25.698+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:26.225+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:26.225+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:26.225+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:26.225+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:26.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:26.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:26.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:26.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:26.049+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:26.674+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:26.674+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:26.674+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:26.674+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:27.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:27.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:27.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:27.217+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:27.017+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:27.017+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:27.017+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:27.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:27.017+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:27.984 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:58:27.532+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 4290 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:58:27.984 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:27.649+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:27.985 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:27.649+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:27.985 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:27.649+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:27.985 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:27.650+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:28.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:27.984+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:28.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:27.984+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:28.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:27.984+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:28.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:27.984+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:28.214+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:28.214+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:28.214+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:28.214+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:28.657+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:28.657+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:28.657+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:28.657+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:29.025+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:29.025+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:29.025+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:29.025+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:29.219+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:29.219+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:29.219+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:29.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:29.219+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:29.995 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:29.706+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:29.995 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:29.706+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:29.995 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:29.706+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:29.995 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:29.706+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:30.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:30.222+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:30.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:30.222+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:30.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:30.222+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:30.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:30.222+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:30.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:29.995+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:30.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:29.995+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:30.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:29.995+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:30.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:29.995+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:30.991 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:30.725+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:30.991 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:30.725+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:30.991 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:30.725+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:30.991 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:30.726+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:31.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:31.199+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:31.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:31.199+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:31.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:31.199+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:31.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:31.199+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:31.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:30.991+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:31.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:30.991+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:31.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:30.991+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:31.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:30.991+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:31.951 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:31.685+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:31.952 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:31.685+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:31.952 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:31.685+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:31.952 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:31.685+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:32.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:32.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:32.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:32.199+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:32.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:31.951+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:32.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:31.951+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:32.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:31.951+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:32.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:31.951+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:32.929 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:58:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:58:32.533+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 4383 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:58:32.930 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:32.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:32.930 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:32.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:32.930 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:32.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:32.930 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:32.707+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:33.203 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:32.929+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:33.203 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:32.929+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:33.203 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:32.929+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:33.203 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:32.929+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:33.203+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:33.203+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:33.203+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:33.203+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:33.950 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:33.680+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:33.950 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:33.680+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:33.950 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:33.680+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:33.950 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:33.680+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:34.187+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:34.187+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:34.187+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:34.187+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:34.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:33.949+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:34.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:33.949+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:34.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:33.949+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:34.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:33.949+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:34.708+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:34.708+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:34.708+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:34.708+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:35.149+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:35.149+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:35.149+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:35.149+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:35.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:34.974+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:35.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:34.974+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:35.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:34.974+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:35.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:34.974+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:35.751+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:35.751+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:35.751+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:35.751+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:36.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:36.164+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:36.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:36.164+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:36.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:36.164+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:36.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:36.164+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:36.006+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:36.006+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:36.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:36.006+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:36.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:36.006+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:36.716+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:36.716+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:36.716+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:36.716+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:37.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:36.963+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:37.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:36.963+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:37.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:36.963+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:37.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:36.963+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:37.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:37.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:37.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:37.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:37.196+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:37.930 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:58:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:58:37.534+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 4488 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:58:37.931 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:37.670+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:37.931 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:37.670+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:37.931 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:37.670+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:37.931 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:37.670+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:38.240 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:37.930+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:38.240 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:37.930+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:38.240 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:37.930+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:38.240 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:37.931+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:38.239+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:38.239+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:38.239+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:38.240+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:38.939 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:38.638+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:38.939 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:38.638+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:38.939 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:38.638+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:38.939 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:38.638+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:39.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:38.939+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:39.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:38.939+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:39.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:38.939+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:39.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:38.939+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:39.669 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:39.254+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:39.669 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:39.254+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:39.669 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:39.254+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:39.669 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:39.254+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:39.942 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:39.669+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:39.942 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:39.669+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:39.942 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:39.669+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:39.942 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:39.669+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:40.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:39.942+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:40.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:39.942+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:40.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:39.942+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:40.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:39.942+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:40.628 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:40.299+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:40.628 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:40.299+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:40.628 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:40.299+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:40.628 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:40.299+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:40.921+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:40.921+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:40.921+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:40.921+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:41.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:40.689+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:41.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:40.689+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:41.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:40.689+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:41.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:40.689+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:41.655 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:41.270+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:41.655 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:41.270+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:41.655 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:41.270+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:41.655 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:41.270+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:41.923 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:41.739+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:41.923 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:41.739+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:41.923 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:41.739+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:41.923 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:41.739+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:42.235 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:41.923+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:42.235 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:41.923+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:42.235 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:41.923+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:42.235 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:41.923+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:42.235+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:42.235+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:42.235+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:42.235+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:42.893 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:58:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:58:42.536+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 4598 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:58:42.893 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:42.772+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:42.894 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:42.772+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:42.894 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:42.772+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:42.894 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:42.772+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:43.199 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:42.893+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:43.200 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:42.893+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:43.200 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:42.893+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:43.200 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:42.893+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:43.200+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:43.200+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:43.200+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:43.200+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:43.893 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:43.784+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:43.894 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:43.784+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:43.894 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:43.784+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:43.894 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:43.784+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:44.156 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:43.893+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:44.156 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:43.893+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:44.156 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:43.893+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:44.156 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:43.893+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:44.156+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:44.156+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:44.156+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:44.156+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:45.139 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:44.859+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:45.139 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:44.859+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:45.139 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:44.859+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:45.140 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:44.859+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:45.140 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:44.803+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:45.140 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:44.803+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:45.140 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:44.803+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:45.140 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:44.803+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:45.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:45.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:45.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:45.139+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:46.165 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:45.826+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:46.166 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:45.826+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:46.166 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:45.826+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:46.166 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:45.826+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:46.166 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:45.811+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:46.166 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:45.811+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:46.166 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:45.811+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:46.166 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:45.811+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:46.165+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:46.165+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:46.165+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:46.165+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:47.180 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:46.845+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:47.180 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:46.845+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:47.180 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:46.845+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:47.180 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:46.845+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:47.181 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:46.770+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:47.181 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:46.770+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:47.181 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:46.770+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:47.181 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:46.770+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:47.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:47.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:47.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:47.180+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:47.847 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:58:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:58:47.537+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 4727 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:58:47.847 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:47.774+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:47.848 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:47.774+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:47.848 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:47.774+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:47.848 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:47.774+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:48.208 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:47.846+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:48.208 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:47.847+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:48.209 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:47.847+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:48.209 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:47.847+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:48.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:48.208+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:48.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:48.208+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:48.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:48.208+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:48.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:48.208+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:49.184 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:48.811+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:49.184 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:48.811+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:49.184 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:48.811+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:49.185 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:48.811+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:49.185 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:48.806+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:49.185 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:48.806+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:49.185 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:48.806+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:49.185 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:48.806+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:49.184+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:49.184+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:49.184+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:49.184+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:50.183 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:49.794+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:50.183 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:49.794+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:50.183 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:49.794+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:50.183 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:49.795+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:50.183 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:49.759+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:50.183 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:49.759+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:50.184 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:49.759+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:50.184 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:49.759+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:50.183+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:50.183+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:50.183+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:50.183+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:50.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:50.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:50.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:50.796+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:51.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:50.714+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:51.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:50.714+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:51.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:50.714+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:51.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:50.714+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:51.219+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:51.219+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:51.219+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:51.220+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:51.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:51.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:51.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:51.761+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:51.725+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:51.725+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:51.725+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:51.725+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:52.539 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:52.264+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:52.539 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:52.264+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:52.540 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:52.264+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:52.540 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:52.264+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:52.916 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:58:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:58:52.538+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 4821 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:58:52.916 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:52.725+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:52.916 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:52.725+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:52.917 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:52.725+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:52.917 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:52.725+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:52.917 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:52.719+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:52.917 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:52.719+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:52.917 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:52.719+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:52.917 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:52.719+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:53.545 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:53.259+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:53.545 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:53.259+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:53.545 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:53.259+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:53.546 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:53.259+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:53.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:53.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:53.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:53.735+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:53.690+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:53.690+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:53.690+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:54.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:53.690+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:54.648 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:54.289+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:54.648 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:54.289+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:54.648 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:54.289+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:54.648 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:54.289+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:54.733+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:54.733+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:54.733+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:54.733+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:55.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:54.648+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:55.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:54.648+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:55.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:54.648+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:55.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:54.648+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:55.603 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:55.261+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:55.603 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:55.261+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:55.603 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:55.261+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:55.603 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:55.262+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:55.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:55.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:55.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:55.776+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:55.602+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:55.602+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:55.602+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:55.602+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:56.567 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:56.277+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:56.567 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:56.277+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:56.567 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:56.277+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:56.567 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:56.277+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:56.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:56.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:56.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:56.727+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:57.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:56.565+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:57.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:56.565+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:57.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:56.565+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:57.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:56.566+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:57.567 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:58:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:58:57.538+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 4916 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:58:57.567 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:57.546+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:57.567 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:57.546+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:57.567 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:57.546+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:57.567 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:57.547+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:57.568 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:57.313+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:57.568 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:57.313+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:57.568 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:57.313+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:57.568 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:57.313+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:57.699+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:57.700+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:57.700+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:57.700+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:58.736 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:58.325+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:58.736 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:58.325+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:58.737 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:58.325+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:58.737 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:58.325+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:58.737 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:58.498+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:58.737 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:58.498+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:58.737 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:58.498+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:58.737 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:58.498+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:59.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:58.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:58.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:58.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:58:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:58.735+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:59.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:59.353+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:59.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:59.353+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:59.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:59.353+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:58:59.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:58:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:58:59.353+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:58:59.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:59.535+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:59.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:59.535+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:59.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:59.535+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:58:59.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:58:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:58:59.535+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:00.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:59.749+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:59.749+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:59.749+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:58:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:58:59.750+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:00.709 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:00.368+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:00.709 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:00.368+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:00.709 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:00.368+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:00.709 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:00.368+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:00.709 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:00.574+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:00.709 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:00.574+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:00.709 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:00.574+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:00.710 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:00.574+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:00.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:00.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:00.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:00.708+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:01.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:01.366+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:01.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:01.366+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:01.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:01.366+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:01.658 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:01.366+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:01.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:01.531+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:01.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:01.531+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:01.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:01.531+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:01.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:01.531+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:01.717+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:01.717+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:01.717+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:01.717+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:02.669 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:59:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:59:02.540+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 5012 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:59:02.670 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:02.485+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:02.670 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:02.485+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:02.670 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:02.485+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:02.670 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:02.485+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:02.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:02.319+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:02.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:02.319+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:02.671 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:02.319+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:02.671 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:02.320+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:02.675+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:02.675+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:02.675+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:02.675+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:03.668 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:03.307+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:03.668 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:03.307+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:03.668 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:03.307+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:03.669 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:03.307+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:03.669 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:03.486+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:03.669 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:03.486+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:03.669 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:03.486+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:03.669 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:03.486+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:03.667+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:03.667+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:03.667+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:03.667+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:04.666 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:04.301+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:04.667 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:04.301+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:04.667 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:04.301+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:04.667 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:04.301+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:04.667 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:04.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:04.667 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:04.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:04.667 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:04.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:04.667 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:04.465+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:04.666+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:04.666+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:04.666+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:04.666+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:05.491 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:05.269+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:05.492 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:05.269+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:05.492 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:05.269+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:05.492 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:05.269+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:05.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:05.491+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:05.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:05.491+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:05.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:05.491+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:05.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:05.491+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:05.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:05.620+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:05.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:05.620+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:05.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:05.620+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:05.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:05.621+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:06.595 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:06.291+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:06.595 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:06.291+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:06.595 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:06.291+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:06.596 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:06.291+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:06.596 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:06.517+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:06.596 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:06.517+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:06.596 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:06.518+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:06.596 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:06.518+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:06.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:06.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:06.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:06.594+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:07.565 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:59:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:59:07.541+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 5106 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:59:07.565 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:07.313+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:07.565 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:07.313+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:07.565 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:07.313+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:07.565 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:07.313+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:07.566 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:07.556+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:07.566 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:07.556+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:07.566 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:07.556+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:07.566 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:07.556+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:07.570+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:07.570+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:07.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:07.571+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:08.604 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:08.333+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:08.604 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:08.333+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:08.604 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:08.333+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:08.604 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:08.333+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:08.604 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:08.577+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:08.604 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:08.577+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:08.605 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:08.577+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:08.605 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:08.577+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:08.603+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:08.603+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:08.603+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:08.603+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:09.379+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:09.379+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:09.379+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:09.379+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:09.575+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:09.575+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:09.575+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:09.576+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:09.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:09.584+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:09.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:09.584+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:09.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:09.584+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:09.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:09.584+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:10.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:10.567+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:10.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:10.567+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:10.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:10.567+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:10.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:10.567+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:10.375+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:10.375+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:10.375+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:10.375+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:10.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:10.567+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:10.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:10.567+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:10.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:10.567+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:10.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:10.568+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:11.608 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:11.388+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:11.608 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:11.388+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:11.608 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:11.388+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:11.608 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:11.388+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:11.609 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:11.576+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:11.609 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:11.576+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:11.609 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:11.576+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:11.609 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:11.576+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:11.607+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:11.607+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:11.607+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:12.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:11.607+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:12.751 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:59:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:59:12.543+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 5201 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:59:12.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:12.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:12.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:12.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:12.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:12.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:12.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:12.606+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:12.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:12.433+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:12.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:12.433+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:12.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:12.433+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:12.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:12.434+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:12.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:12.580+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:12.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:12.580+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:12.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:12.580+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:12.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:12.580+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:13.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:13.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:13.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:13.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:13.612+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:13.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:13.587+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:13.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:13.587+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:13.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:13.587+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:13.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:13.587+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:13.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:13.480+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:13.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:13.480+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:13.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:13.480+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:13.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:13.481+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:14.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:14.473+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:14.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:14.473+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:14.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:14.473+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:14.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:14.474+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:14.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:14.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:14.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:14.572+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:14.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:14.576+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:14.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:14.576+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:14.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:14.576+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:14.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:14.576+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:15.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:15.562+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:15.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:15.562+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:15.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:15.562+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:15.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:15.562+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:15.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:15.477+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:15.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:15.477+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:15.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:15.477+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:15.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:15.477+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:15.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:15.565+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:15.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:15.565+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:15.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:15.565+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:15.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:15.566+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:16.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:16.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:16.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:16.559+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:16.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:16.518+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:16.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:16.518+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:16.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:16.518+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:16.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:16.519+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:16.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:16.498+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:16.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:16.498+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:16.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:16.498+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:16.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:16.499+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:17.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:59:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:59:17.545+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 5296 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:59:17.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:17.476+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:17.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:17.476+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:17.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:17.476+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:17.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:17.477+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:17.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:17.537+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:17.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:17.537+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:17.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:17.537+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:17.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:17.538+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:17.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:17.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:17.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:17.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:17.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:17.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:17.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:17.586+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:18.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:18.491+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:18.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:18.491+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:18.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:18.491+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:18.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:18.491+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:18.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:18.548+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:18.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:18.548+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:18.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:18.548+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:18.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:18.548+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:18.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:18.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:18.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:18.591+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:19.528+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:19.528+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:19.528+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:19.529+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:19.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:19.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:19.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:19.591+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:20.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:19.511+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:20.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:19.511+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:20.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:19.511+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:20.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:19.512+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:20.558+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:20.558+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:20.558+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:20.559+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:20.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:20.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:20.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:20.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:20.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:20.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:20.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:20.465+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:20.623+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:20.623+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:20.623+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:20.623+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:21.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:21.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:21.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:21.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:21.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:21.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:21.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:21.517+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:21.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:21.450+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:21.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:21.450+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:21.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:21.450+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:21.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:21.450+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:21.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:21.587+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:21.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:21.587+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:21.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:21.587+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:21.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:21.588+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:22.673 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:59:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:59:22.546+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 5392 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:59:22.673 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:22.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:22.673 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:22.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:22.674 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:22.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:22.674 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:22.618+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:22.674 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:22.412+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:22.674 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:22.412+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:22.674 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:22.412+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:22.674 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:22.412+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:22.674 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:22.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:22.674 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:22.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:22.675 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:22.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:22.675 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:22.541+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:23.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:23.523+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:23.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:23.523+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:23.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:23.523+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:23.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:23.524+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:23.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:23.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:23.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:23.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:23.585+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:23.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:23.397+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:23.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:23.397+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:23.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:23.397+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:23.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:23.397+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:24.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:24.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:24.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:24.571+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:24.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:24.355+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:24.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:24.355+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:24.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:24.355+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:24.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:24.355+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:24.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:24.511+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:24.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:24.511+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:24.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:24.511+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:24.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:24.511+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:25.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:25.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:25.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:25.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:25.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:25.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:25.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:25.400+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:25.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:25.489+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:25.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:25.489+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:25.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:25.489+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:25.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:25.489+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:25.578+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:25.578+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:25.578+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:25.578+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:26.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:26.449+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:26.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:26.449+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:26.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:26.449+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:26.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:26.450+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:26.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:26.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:26.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:26.611+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:26.381+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:26.381+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:26.381+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:26.382+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:27.626 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:59:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:59:27.547+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 5486 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:59:27.626 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:27.416+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:27.626 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:27.416+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:27.626 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:27.416+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:27.627 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:27.416+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:27.627 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:27.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:27.627 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:27.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:27.627 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:27.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:27.627 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:27.356+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:28.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:27.625+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:28.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:27.625+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:28.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:27.625+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:28.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:27.626+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:28.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:28.422+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:28.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:28.422+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:28.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:28.422+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:28.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:28.422+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:28.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:28.357+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:28.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:28.357+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:28.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:28.357+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:28.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:28.357+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:28.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:28.588+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:28.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:28.588+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:28.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:28.588+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:28.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:28.589+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:29.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:29.347+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:29.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:29.347+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:29.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:29.347+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:29.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:29.347+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:29.632 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:29.459+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:29.633 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:29.459+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:29.633 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:29.459+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:29.633 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:29.460+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:29.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:29.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:29.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:29.632+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:30.654 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:30.421+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:30.654 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:30.421+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:30.654 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:30.421+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:30.654 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:30.421+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:30.654 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:30.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:30.654 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:30.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:30.655 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:30.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:30.655 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:30.374+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:30.653+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:30.653+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:30.653+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:30.654+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:31.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:31.331+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:31.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:31.331+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:31.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:31.331+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:31.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:31.331+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:31.659 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:31.382+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:31.659 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:31.382+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:31.659 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:31.382+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:31.659 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:31.383+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:32.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:31.674+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:31.674+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:31.674+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:31.675+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:32.559 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:59:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:59:32.548+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 5581 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:59:32.560 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:32.300+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:32.560 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:32.300+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:32.560 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:32.300+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:32.560 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:32.300+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:32.560 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:32.396+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:32.561 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:32.396+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:32.561 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:32.396+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:32.561 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:32.397+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:32.851 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:32.654+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:32.852 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:32.654+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:32.852 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:32.654+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:32.852 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:32.654+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:33.633 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:33.392+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:33.633 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:33.392+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:33.633 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:33.392+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:33.633 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:33.393+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:33.633 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:33.318+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:33.634 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:33.318+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:33.634 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:33.318+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:33.634 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:33.318+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:33.632+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:33.632+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:33.632+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:33.633+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:34.625 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:34.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:34.625 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:34.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:34.625 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:34.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:34.625 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:34.424+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:34.626 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:34.333+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:34.626 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:34.333+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:34.626 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:34.333+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:34.626 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:34.333+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:34.625+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:34.625+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:34.625+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:34.625+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:35.662 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:35.395+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:35.662 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:35.395+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:35.662 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:35.395+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:35.662 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:35.395+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:35.662 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:35.341+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:35.663 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:35.341+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:35.663 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:35.341+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:35.663 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:35.341+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:35.923 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:35.661+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:35.924 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:35.661+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:35.924 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:35.661+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:35.924 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:35.661+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:36.706 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:36.347+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:36.706 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:36.347+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:36.706 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:36.347+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:36.706 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:36.347+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:36.707 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:36.339+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:36.707 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:36.339+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:36.707 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:36.339+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:36.707 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:36.339+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:36.706+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:36.706+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:36.706+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:36.706+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:37.550 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:37.298+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:37.550 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:37.298+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:37.551 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:37.298+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:37.551 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:37.298+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:37.551 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:37.382+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:37.551 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:37.382+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:37.551 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:37.382+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:37.551 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:37.383+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:38.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:59:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:59:37.550+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 5675 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:59:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:37.677+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:37.677+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:38.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:37.677+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:38.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:37.678+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:38.648 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:38.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:38.648 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:38.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:38.648 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:38.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:38.648 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:38.423+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:38.648 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:38.294+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:38.648 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:38.294+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:38.648 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:38.294+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:38.648 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:38.295+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:39.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:38.647+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:38.647+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:38.647+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:38.648+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:39.682 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:39.417+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:39.682 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:39.417+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:39.683 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:39.417+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:39.683 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:39.418+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:39.683 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:39.267+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:39.683 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:39.267+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:39.683 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:39.267+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:39.683 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:39.267+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:39.682+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:39.682+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:39.682+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:39.682+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:40.696 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:40.387+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:40.696 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:40.387+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:40.696 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:40.387+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:40.696 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:40.387+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:40.696 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:40.280+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:40.696 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:40.280+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:40.696 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:40.280+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:40.697 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:40.281+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:40.695+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:40.695+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:40.695+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:40.696+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:41.658 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:41.364+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:41.659 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:41.364+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:41.659 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:41.364+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:41.659 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:41.365+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:41.659 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:41.261+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:41.659 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:41.261+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:41.659 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:41.261+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:41.659 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:41.261+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:41.708+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:41.708+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:41.708+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:41.708+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:42.551 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:42.263+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:42.552 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:42.263+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:42.552 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:42.263+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:42.552 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:42.263+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:42.552 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:42.397+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:42.552 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:42.397+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:42.552 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:42.397+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:42.552 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:42.398+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:43.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:59:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:59:42.551+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 5770 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:59:43.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:42.670+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:43.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:42.670+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:43.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:42.670+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:43.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:42.670+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:43.410+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:43.410+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:43.410+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:43.411+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:43.242+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:43.242+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:43.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:43.242+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:43.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:43.242+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:43.700+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:43.700+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:43.700+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:43.700+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:44.390+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:44.390+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:44.390+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:44.390+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:44.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:44.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:44.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:44.209+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:44.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:44.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:44.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:44.736+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:45.393 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:45.161+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:45.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:45.161+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:45.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:45.161+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:45.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:45.161+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:45.710 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:45.393+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:45.710 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:45.393+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:45.710 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:45.393+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:45.710 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:45.394+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:45.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:45.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:45.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:45.710+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:46.176+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:46.176+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:46.176+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:46.176+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:46.387+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:46.387+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:46.387+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:46.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:46.388+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:47.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:46.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:46.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:46.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:46.662+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:47.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:47.142+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:47.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:47.142+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:47.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:47.142+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:47.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:47.142+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:47.679 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:59:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:59:47.552+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 5864 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:59:47.679 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:47.393+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:47.679 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:47.393+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:47.679 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:47.393+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:47.680 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:47.394+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:47.696+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:47.696+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:47.696+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:47.697+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:48.357 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:48.102+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:48.357 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:48.102+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:48.357 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:48.102+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:48.357 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:48.102+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:48.738 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:48.357+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:48.738 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:48.357+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:48.738 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:48.357+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:48.738 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:48.357+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:48.738+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:48.738+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:48.738+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:48.738+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:49.392 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:49.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:49.392 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:49.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:49.392 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:49.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:49.392 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:49.116+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:49.735 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:49.391+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:49.735 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:49.391+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:49.735 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:49.391+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:49.735 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:49.392+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:49.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:49.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:49.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:49.735+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:50.410 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:50.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:50.410 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:50.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:50.410 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:50.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:50.410 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:50.141+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:50.719 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:50.409+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:50.719 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:50.409+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:50.719 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:50.409+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:50.719 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:50.410+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:50.719+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:50.719+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:50.719+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:50.719+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:51.419+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:51.419+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:51.419+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:51.420+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:51.177+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:51.177+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:51.177+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:51.177+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:51.692+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:51.692+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:51.692+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:51.692+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:52.372+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:52.372+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:52.372+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:52.372+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:52.150+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:52.150+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:52.150+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:52.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:52.150+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:53.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:59:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:59:52.554+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 5959 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:59:53.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:52.672+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:53.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:52.672+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:53.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:52.672+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:53.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:52.672+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:53.138+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:53.138+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:53.138+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:53.138+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:53.355+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:53.355+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:53.355+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:53.355+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:53.694+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:53.694+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:53.694+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:53.694+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:54.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:54.160+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:54.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:54.160+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:54.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:54.160+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:54.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:54.160+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:54.338+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:54.338+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:54.338+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:54.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:54.338+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:54.728+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:54.728+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:54.728+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:54.729+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:55.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:55.198+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:55.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:55.198+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:55.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:55.198+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:55.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:55.198+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:55.386+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:55.386+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:55.386+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:55.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:55.387+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:55.747+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:55.747+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:55.747+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:55.747+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:56.437 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:56.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:56.437 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:56.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:56.438 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:56.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:56.438 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:56.178+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:56.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:56.437+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:56.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:56.437+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:56.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:56.437+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:56.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:56.437+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:57.161 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:56.780+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:57.161 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:56.780+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:57.161 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:56.780+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:57.161 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:56.780+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:57.428 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:57.161+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:57.428 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:57.161+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:57.428 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:57.161+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:57.428 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:57.161+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:57.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 20:59:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T20:59:57.555+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 6053 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T20:59:57.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:57.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:57.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:57.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:57.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:57.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:57.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:57.452+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:58.010 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:57.758+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:58.010 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:57.758+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:58.010 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:57.758+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:58.010 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:57.758+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:58.444 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:58.147+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:58.445 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:58.147+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:58.445 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:58.147+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:58.445 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:58.148+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:58.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:58.444+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:58.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:58.444+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:58.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:58.444+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:58.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:58.445+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:59.139 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:58.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:59.139 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:58.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:59.140 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:58.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T20:59:59.140 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:58.752+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:59.448 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:59.139+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:59.449 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:59.139+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:59.449 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:59.139+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T20:59:59.449 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 20:59:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T20:59:59.139+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T20:59:59.705 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:59.448+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:59.706 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:59.448+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:59.706 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:59.448+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T20:59:59.706 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 20:59:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T20:59:59.448+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:00.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:59.705+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:59.705+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:59.705+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 20:59:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T20:59:59.705+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:00.457 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:00.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:00.457 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:00.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:00.458 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:00.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:00.458 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:00.129+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:00.710 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:00.457+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:00.711 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:00.457+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:00.711 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:00.457+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:00.711 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:00.457+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:00.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:00.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:00.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:00.710+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:01.419 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:01.167+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:01.419 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:01.167+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:01.419 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:01.167+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:01.420 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:01.167+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:01.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:01.419+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:01.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:01.419+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:01.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:01.419+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:01.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:01.419+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:02.196 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:01.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:02.196 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:01.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:02.196 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:01.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:02.196 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:01.752+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:02.195+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:02.195+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:02.195+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:02.196+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:02.417+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:02.417+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:02.417+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:02.418+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:03.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:00:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:00:02.556+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 6150 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:00:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:02.759+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:02.759+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:02.759+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:02.759+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:03.383+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:03.383+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:03.383+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:03.383+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:03.188+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:03.188+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:03.188+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:03.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:03.188+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:03.741+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:03.741+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:03.741+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:03.741+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:04.334+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:04.334+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:04.334+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:04.335+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:04.224+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:04.224+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:04.224+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:04.224+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:04.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:04.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:04.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:04.710+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:05.316+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:05.316+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:05.316+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:05.316+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:05.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:05.237+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:05.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:05.237+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:05.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:05.237+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:05.237+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:05.713+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:05.713+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:05.713+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:05.713+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:06.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:06.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:06.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:06.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:06.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:06.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:06.197+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:06.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:06.281+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:06.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:06.281+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:06.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:06.281+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:06.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:06.281+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:06.729+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:06.729+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:06.729+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:06.730+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:07.236+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:07.236+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:07.236+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:07.236+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:07.313+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:07.313+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:07.313+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:07.313+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:08.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:00:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:00:07.557+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 6243 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:00:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:07.754+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:07.754+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:07.754+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:07.754+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:08.276+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:08.276+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:08.276+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:08.276+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:08.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:08.264+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:08.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:08.264+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:08.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:08.264+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:08.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:08.264+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:09.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:08.793+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:09.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:08.793+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:09.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:08.793+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:09.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:08.793+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:09.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:09.271+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:09.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:09.271+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:09.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:09.271+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:09.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:09.271+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:09.325+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:09.325+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:09.325+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:09.326+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:10.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:09.834+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:10.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:09.834+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:10.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:09.834+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:10.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:09.834+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:10.286+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:10.286+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:10.286+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:10.286+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:10.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:10.257+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:10.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:10.257+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:10.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:10.257+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:10.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:10.257+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:11.217 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:10.874+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:11.217 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:10.874+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:11.217 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:10.874+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:11.217 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:10.874+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:11.216+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:11.216+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:11.216+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:11.217+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:11.257+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:11.257+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:11.257+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:11.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:11.258+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:12.190 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:11.918+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:12.190 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:11.918+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:12.191 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:11.918+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:12.191 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:11.918+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:12.280+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:12.280+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:12.280+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:12.281+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:12.190+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:12.190+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:12.190+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:12.190+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:12.902 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:00:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:00:12.559+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 6339 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:00:13.206 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:12.902+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:13.206 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:12.902+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:13.206 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:12.902+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:13.206 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:12.902+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:13.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:13.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:13.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:13.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:13.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:13.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:13.206+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:13.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:13.303+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:13.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:13.303+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:13.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:13.303+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:13.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:13.303+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:14.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:13.935+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:14.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:13.935+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:14.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:13.935+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:14.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:13.936+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:14.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:14.170+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:14.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:14.170+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:14.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:14.170+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:14.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:14.170+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:14.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:14.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:14.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:14.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:14.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:14.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:14.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:14.337+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:15.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:15.194+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:15.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:15.194+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:15.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:15.194+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:15.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:15.194+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:15.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:14.954+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:15.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:14.954+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:15.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:14.954+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:15.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:14.954+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:15.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:15.351+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:15.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:15.351+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:15.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:15.351+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:15.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:15.351+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:16.210 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:15.920+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:16.210 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:15.920+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:16.210 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:15.920+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:16.210 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:15.921+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:16.345+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:16.345+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:16.345+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:16.345+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:16.210+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:16.210+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:16.210+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:16.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:16.210+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:17.190 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:16.878+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:17.190 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:16.878+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:17.190 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:16.878+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:17.190 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:16.878+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:17.369+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:17.369+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:17.369+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:17.369+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:17.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:17.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:17.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:17.189+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:17.924 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:00:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:00:17.559+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 6433 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:00:18.181 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:17.923+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:18.181 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:17.923+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:18.181 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:17.923+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:18.181 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:17.923+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:18.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:18.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:18.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:18.180+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:18.352+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:18.352+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:18.352+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:18.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:18.352+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:19.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:18.921+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:19.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:18.921+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:19.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:18.921+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:19.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:18.921+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:19.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:19.157+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:19.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:19.157+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:19.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:19.157+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:19.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:19.158+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:19.342+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:19.342+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:19.342+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:19.343+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:20.200 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:19.887+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:20.200 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:19.887+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:20.200 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:19.887+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:20.200 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:19.887+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:20.198+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:20.198+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:20.198+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:20.199+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:20.384+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:20.384+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:20.384+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:20.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:20.384+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:21.221 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:20.896+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:21.221 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:20.896+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:21.221 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:20.896+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:21.221 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:20.896+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:21.220+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:21.220+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:21.220+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:21.220+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:21.360+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:21.360+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:21.360+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:21.360+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:22.227 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:21.931+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:22.227 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:21.931+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:22.227 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:21.931+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:22.227 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:21.931+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:22.401+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:22.401+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:22.401+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:22.402+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:22.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:22.226+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:22.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:22.226+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:22.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:22.226+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:22.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:22.226+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:22.940 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:00:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:00:22.561+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 6530 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:00:23.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:22.940+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:23.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:22.940+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:23.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:22.940+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:23.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:22.940+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:23.552 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:23.411+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:23.552 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:23.411+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:23.552 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:23.411+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:23.552 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:23.411+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:23.553 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:23.266+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:23.553 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:23.266+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:23.553 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:23.266+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:23.553 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:23.266+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:24.242 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:23.917+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:24.242 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:23.917+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:24.242 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:23.917+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:24.242 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:23.917+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:24.241+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:24.241+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:24.241+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:24.241+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:24.414+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:24.414+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:24.414+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:24.415+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:25.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:24.965+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:25.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:24.965+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:25.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:24.965+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:25.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:24.965+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:25.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:25.193+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:25.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:25.193+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:25.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:25.193+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:25.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:25.193+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:25.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:25.381+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:25.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:25.381+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:25.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:25.381+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:25.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:25.382+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:26.234 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:25.947+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:26.234 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:25.947+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:26.234 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:25.947+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:26.234 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:25.947+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:26.232+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:26.232+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:26.232+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:26.233+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:26.410+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:26.410+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:26.410+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:26.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:26.410+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:27.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:26.986+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:27.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:26.986+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:27.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:26.986+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:27.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:26.987+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:27.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:27.201+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:27.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:27.201+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:27.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:27.201+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:27.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:27.201+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:27.692 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:00:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:00:27.562+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 6623 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:00:27.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:27.407+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:27.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:27.407+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:27.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:27.407+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:27.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:27.407+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:28.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:28.193+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:28.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:28.193+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:28.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:28.193+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:28.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:28.193+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:28.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:27.995+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:28.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:27.996+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:28.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:27.996+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:28.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:27.996+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:28.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:28.404+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:28.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:28.404+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:28.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:28.404+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:28.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:28.404+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:29.400 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:29.004+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:29.400 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:29.004+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:29.400 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:29.004+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:29.400 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:29.004+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:29.400 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:29.225+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:29.400 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:29.225+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:29.400 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:29.225+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:29.401 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:29.225+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:29.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:29.399+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:29.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:29.399+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:29.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:29.399+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:29.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:29.399+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:30.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:30.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:30.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:30.197+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:30.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:29.971+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:30.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:29.971+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:30.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:29.971+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:30.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:29.971+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:30.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:30.422+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:30.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:30.422+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:30.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:30.422+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:30.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:30.422+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:31.220 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:30.969+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:31.220 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:30.969+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:31.220 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:30.969+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:31.220 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:30.969+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:31.463+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:31.463+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:31.463+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:31.463+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:31.222+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:31.222+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:31.222+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:31.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:31.222+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:32.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:31.979+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:32.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:31.979+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:32.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:31.979+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:32.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:31.979+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:32.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:32.191+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:32.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:32.191+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:32.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:32.191+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:32.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:32.191+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:32.692 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:00:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:00:32.564+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 6719 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:00:32.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:32.418+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:32.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:32.418+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:32.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:32.418+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:32.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:32.418+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:33.222 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:32.962+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:33.222 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:32.962+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:33.222 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:32.962+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:33.222 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:32.963+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:33.221+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:33.221+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:33.221+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:33.222+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:33.417+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:33.417+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:33.417+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:33.418+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:34.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:33.972+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:34.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:33.972+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:34.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:33.972+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:34.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:33.973+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:34.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:34.183+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:34.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:34.183+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:34.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:34.183+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:34.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:34.184+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:34.455+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:34.455+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:34.455+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:34.456+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:35.477 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:35.009+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:35.477 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:35.009+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:35.477 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:35.009+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:35.477 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:35.010+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:35.478 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:35.150+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:35.478 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:35.150+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:35.478 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:35.150+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:35.478 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:35.151+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:35.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:35.476+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:35.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:35.477+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:35.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:35.477+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:35.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:35.477+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:36.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:36.033+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:36.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:36.033+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:36.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:36.033+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:36.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:36.034+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:36.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:36.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:36.178+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:36.178+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:36.996 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:36.517+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:36.996 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:36.517+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:36.997 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:36.517+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:36.997 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:36.518+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:37.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:36.996+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:37.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:36.996+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:37.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:36.996+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:37.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:36.997+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:37.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:37.150+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:37.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:37.150+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:37.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:37.150+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:37.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:37.150+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:37.692 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:00:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:00:37.566+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 6813 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:00:37.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:37.530+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:37.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:37.530+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:37.693 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:37.530+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:37.693 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:37.530+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:38.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:38.029+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:38.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:38.029+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:38.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:38.029+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:38.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:38.029+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:38.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:38.109+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:38.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:38.109+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:38.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:38.109+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:38.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:38.109+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:38.988 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:38.543+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:38.988 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:38.543+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:38.988 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:38.543+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:38.988 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:38.543+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:39.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:38.987+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:39.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:38.987+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:39.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:38.987+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:39.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:38.988+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:39.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:39.155+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:39.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:39.155+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:39.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:39.155+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:39.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:39.155+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:39.951 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:39.551+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:39.951 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:39.551+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:39.951 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:39.551+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:39.951 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:39.552+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:40.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:39.950+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:40.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:39.950+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:40.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:39.950+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:40.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:39.951+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:40.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:40.163+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:40.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:40.163+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:40.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:40.163+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:40.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:40.163+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:40.521+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:40.521+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:40.521+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:40.521+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:41.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:41.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:41.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:41.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:41.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:41.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:41.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:41.141+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:41.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:40.923+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:41.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:40.923+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:41.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:40.923+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:41.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:40.923+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:41.903 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:41.549+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:41.904 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:41.549+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:41.904 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:41.549+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:41.904 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:41.550+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:42.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:41.903+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:42.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:41.903+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:42.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:41.903+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:42.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:41.904+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:42.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:42.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:42.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:42.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:42.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:42.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:42.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:42.115+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:42.909 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:00:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:00:42.567+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 6910 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:00:42.909 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:42.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:42.909 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:42.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:42.909 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:42.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:42.909 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:42.521+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:43.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:42.909+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:43.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:42.909+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:43.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:42.909+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:43.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:42.909+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:43.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:43.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:43.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:43.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:43.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:43.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:43.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:43.113+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:43.875 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:43.536+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:43.875 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:43.536+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:43.875 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:43.536+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:43.875 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:43.536+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:44.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:44.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:44.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:44.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:44.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:44.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:44.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:44.088+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:44.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:43.874+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:44.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:43.874+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:44.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:43.875+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:44.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:43.875+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:44.871 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:44.538+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:44.871 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:44.538+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:44.871 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:44.538+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:44.871 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:44.538+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:45.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:44.870+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:45.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:44.870+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:45.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:44.870+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:45.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:44.871+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:45.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:45.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:45.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:45.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:45.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:45.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:45.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:45.049+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:45.859 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:45.517+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:45.859 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:45.517+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:45.859 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:45.517+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:45.859 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:45.518+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:46.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:45.858+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:46.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:45.858+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:46.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:45.858+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:46.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:45.859+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:46.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:46.084+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:46.255 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:46.084+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:46.255 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:46.084+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:46.255 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:46.085+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:46.470+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:46.470+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:46.470+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:46.470+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:47.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:46.848+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:47.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:46.848+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:47.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:46.848+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:47.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:46.848+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:47.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:47.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:47.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:47.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:47.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:47.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:47.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:47.071+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:47.751 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:00:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:00:47.569+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 7003 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:00:47.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:47.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:47.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:47.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:47.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:47.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:47.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:47.451+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:48.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:47.818+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:48.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:47.818+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:48.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:47.818+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:48.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:47.818+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:48.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:48.025+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:48.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:48.025+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:48.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:48.025+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:48.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:48.025+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:48.422+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:48.422+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:48.422+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:48.423+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:48.831+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:48.831+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:48.831+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:48.831+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:49.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:49.044+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:49.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:49.044+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:49.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:49.044+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:49.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:49.044+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:49.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:49.387+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:49.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:49.387+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:49.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:49.387+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:49.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:49.387+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:50.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:50.017+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:50.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:50.017+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:50.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:50.017+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:50.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:50.017+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:50.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:49.799+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:50.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:49.799+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:50.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:49.799+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:50.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:49.799+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:50.437+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:50.437+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:50.437+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:50.438+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:51.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:50.813+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:51.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:50.813+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:51.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:50.813+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:51.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:50.813+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:51.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:51.024+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:51.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:51.024+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:51.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:51.024+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:51.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:51.025+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:51.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:51.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:51.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:51.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:51.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:51.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:51.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:51.423+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:52.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:51.808+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:52.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:51.808+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:52.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:51.808+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:52.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:51.808+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:52.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:52.017+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:52.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:52.017+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:52.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:52.017+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:52.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:52.017+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:52.707 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:00:52.570+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 7099 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:00:52.708 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:52.438+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:52.708 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:52.438+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:52.708 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:52.438+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:52.708 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:52.438+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:52.983 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:52.777+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:52.983 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:52.777+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:52.983 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:52.777+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:52.983 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:52.777+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:53.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:52.982+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:53.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:52.982+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:53.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:52.982+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:53.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:52.982+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:53.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:53.398+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:53.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:53.398+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:53.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:53.398+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:53.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:53.399+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:54.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:53.804+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:54.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:53.804+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:54.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:53.804+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:54.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:53.804+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:54.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:54.029+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:54.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:54.029+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:54.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:54.029+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:54.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:54.029+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:54.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:54.377+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:54.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:54.377+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:54.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:54.377+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:54.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:54.377+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:55.044 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:54.757+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:55.045 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:54.758+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:55.045 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:54.758+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:55.045 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:54.758+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:55.349 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:55.042+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:55.349 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:55.042+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:55.349 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:55.042+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:55.349 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:55.042+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:55.742 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:55.348+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:55.742 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:55.348+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:55.742 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:55.348+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:55.742 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:55.349+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:55.742+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:55.742+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:55.742+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:55.742+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:56.309 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:56.053+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:56.309 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:56.053+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:56.309 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:56.053+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:56.309 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:56.053+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:56.712 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:56.309+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:56.712 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:56.309+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:56.712 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:56.309+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:56.712 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:56.309+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:56.712+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:56.712+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:56.712+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:56.712+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:57.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:57.264+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:57.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:57.264+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:57.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:57.264+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:57.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:57.265+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:57.101+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:57.101+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:57.101+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:57.101+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:58.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:00:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:00:57.572+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 7193 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:00:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:57.704+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:57.704+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:57.704+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:57.705+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:58.256+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:58.256+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:58.256+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:58.257+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:58.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:58.077+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:58.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:58.077+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:58.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:58.077+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:58.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:58.077+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:58.724+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:58.724+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:58.724+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:00:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:58.724+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:59.255+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:59.255+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:59.255+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:00:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:00:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:00:59.256+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:00:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:59.036+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:59.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:59.036+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:59.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:59.036+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:00:59.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:00:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:00:59.036+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:00.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:59.740+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:59.740+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:59.740+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:00:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:00:59.740+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:00.223+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:00.223+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:00.223+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:00.223+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:00.023+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:00.023+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:00.023+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:00.023+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:01.229 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:00.777+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:01.229 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:00.777+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:01.229 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:00.777+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:01.229 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:00.777+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:01.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:00.985+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:01.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:00.985+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:01.230 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:00.985+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:01.230 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:00.986+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:01.235+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:01.235+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:01.235+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:01.236+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:01.939 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:01.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:01.939 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:01.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:01.940 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:01.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:01.940 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:01.761+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:02.195 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:02.007+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:02.196 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:02.007+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:02.196 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:02.007+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:02.196 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:02.007+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:02.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:02.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:02.195+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:02.196+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:03.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:01:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:01:02.573+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 7290 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:01:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:02.782+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:02.782+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:02.782+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:02.783+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:03.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:03.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:03.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:03.181+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:03.039+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:03.039+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:03.039+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:03.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:03.039+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:03.995 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:03.747+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:03.995 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:03.747+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:03.995 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:03.747+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:03.995 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:03.747+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:04.193+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:04.193+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:04.193+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:04.193+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:04.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:03.994+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:04.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:03.994+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:04.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:03.994+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:04.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:03.995+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:04.725+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:04.725+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:04.725+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:04.726+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:05.167+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:05.167+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:05.167+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:05.167+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:05.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:05.039+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:05.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:05.039+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:05.039+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:05.039+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:05.748+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:05.748+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:05.748+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:05.748+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:06.199+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:06.199+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:06.199+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:06.199+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:06.061+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:06.061+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:06.061+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:06.061+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:06.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:06.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:06.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:06.707+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:07.189+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:07.189+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:07.189+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:07.189+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:07.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:07.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:07.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:07.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:07.067+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:08.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:01:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:01:07.575+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 7382 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:01:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:07.690+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:07.690+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:07.690+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:07.690+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:08.284 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:08.232+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:08.284 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:08.232+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:08.284 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:08.232+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:08.285 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:08.233+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:08.285 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:08.030+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:08.285 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:08.030+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:08.285 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:08.030+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:08.285 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:08.031+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:08.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:08.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:08.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:08.662+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:09.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:09.200+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:09.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:09.200+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:09.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:09.200+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:09.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:09.200+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:09.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:09.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:09.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:09.049+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:09.636+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:09.636+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:09.636+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:09.636+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:10.241+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:10.241+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:10.241+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:10.241+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:10.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:10.053+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:10.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:10.053+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:10.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:10.053+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:10.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:10.053+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:10.616+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:10.616+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:10.616+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:10.616+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:11.257+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:11.257+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:11.257+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:11.258+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:11.036+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:11.036+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:11.036+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:11.036+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:11.989 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:11.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:11.989 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:11.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:11.989 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:11.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:11.989 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:11.632+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:12.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:11.989+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:12.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:11.989+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:12.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:11.989+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:12.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:11.989+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:12.576 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:12.290+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:12.577 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:12.290+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:12.577 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:12.290+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:12.577 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:12.290+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:12.945 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:01:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:01:12.576+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 7478 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:01:12.946 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:12.674+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:12.946 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:12.674+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:12.946 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:12.674+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:12.946 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:12.675+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:13.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:12.945+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:13.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:12.945+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:13.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:12.945+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:13.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:12.945+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:13.556 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:13.291+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:13.556 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:13.291+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:13.556 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:13.291+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:13.556 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:13.291+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:13.961 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:13.646+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:13.961 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:13.646+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:13.961 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:13.646+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:13.961 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:13.646+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:14.227 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:13.961+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:14.228 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:13.961+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:14.228 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:13.961+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:14.228 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:13.961+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:14.270+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:14.270+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:14.270+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:14.271+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:14.660+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:14.660+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:14.660+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:14.660+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:15.298 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:15.003+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:15.298 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:15.003+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:15.298 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:15.003+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:15.298 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:15.004+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:15.640 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:15.298+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:15.640 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:15.298+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:15.641 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:15.298+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:15.641 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:15.298+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:15.640+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:15.640+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:15.640+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:15.640+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:16.310 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:16.037+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:16.310 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:16.037+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:16.310 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:16.037+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:16.311 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:16.037+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:16.598 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:16.310+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:16.598 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:16.310+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:16.598 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:16.310+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:16.598 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:16.311+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:16.598+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:16.598+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:16.598+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:16.598+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:17.333 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:17.037+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:17.334 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:17.037+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:17.334 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:17.037+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:17.334 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:17.037+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:17.590 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:01:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:01:17.578+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 7572 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:01:17.590 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:17.333+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:17.590 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:17.333+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:17.590 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:17.333+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:17.590 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:17.334+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:17.590 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:17.550+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:17.591 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:17.550+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:17.591 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:17.550+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:17.591 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:17.551+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:18.367 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:18.045+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:18.367 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:18.045+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:18.367 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:18.045+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:18.368 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:18.045+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:18.718 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:18.367+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:18.719 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:18.367+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:18.719 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:18.367+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:18.719 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:18.367+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:18.719 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:18.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:18.719 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:18.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:18.719 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:18.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:18.719 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:18.590+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:19.330+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:19.330+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:19.330+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:19.331+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:19.082+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:19.082+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:19.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:19.082+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:19.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:19.082+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:19.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:19.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:19.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:19.556+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:20.332+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:20.332+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:20.332+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:20.332+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:20.124+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:20.124+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:20.124+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:20.124+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:20.588+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:20.588+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:20.588+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:20.589+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:21.353+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:21.353+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:21.353+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:21.354+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:21.135+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.546531+0000 front 2023-12-16T20:50:52.546544+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:21.135+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.546491+0000 front 2023-12-16T20:50:52.546566+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:21.135+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.546479+0000 front 2023-12-16T20:50:52.546518+0000 (oldest deadline 2023-12-16T20:51:17.846454+0000) 2023-12-16T21:01:21.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:21.136+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:22.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:21.554+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:21.554+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:21.554+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:21.554+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:22.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:22.088+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:22.327+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:22.327+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:22.327+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:22.328+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:23.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:01:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:01:22.579+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 7668 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:01:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:22.538+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:22.538+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:22.538+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:22.538+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:23.443 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:23.100+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:23.444 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:23.302+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:23.444 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:23.302+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:23.444 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:23.302+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:23.444 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:23.302+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:23.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:23.546+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:23.546+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:23.546+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:23.546+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:24.137+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:24.309+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:24.309+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:24.309+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:24.309+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:24.557+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:24.557+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:24.557+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:24.557+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:25.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:25.136+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:25.263+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:25.263+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:25.263+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:25.263+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:26.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:25.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:25.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:25.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:25.606+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:26.124+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:26.295+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:52.591625+0000 front 2023-12-16T20:50:52.591585+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:26.295+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:52.591564+0000 front 2023-12-16T20:50:52.591637+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:26.295+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:52.591569+0000 front 2023-12-16T20:50:52.591671+0000 (oldest deadline 2023-12-16T20:51:18.491530+0000) 2023-12-16T21:01:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:26.295+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:26.613+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:26.613+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:26.613+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:26.614+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:27.295+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:27.529 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:27.155+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:27.774 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T21:01:27.772+0000 7fe4bff24700 0 monclient(hunting): authenticate timed out after 300 2023-12-16T21:01:27.774 INFO:teuthology.orchestra.run.smithi052.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-16T21:01:27.903 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:01:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:01:27.580+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 7759 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:01:27.903 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:27.589+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:27.903 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:27.589+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:27.904 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:27.589+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:27.904 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:27.589+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:28.099 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2023-12-16T21:01:28.379 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/mon.smithi052/config 2023-12-16T21:01:28.484 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:28.286+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:28.484 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:28.154+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:28.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:28.622+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:28.623+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:28.623+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:28.623+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:29.280+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:29.122+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:29.664+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2023-12-16T20:50:51.988012+0000 front 2023-12-16T20:50:51.988026+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:29.664+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-12-16T20:50:51.987989+0000 front 2023-12-16T20:50:51.988049+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:29.664+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2023-12-16T20:50:51.987941+0000 front 2023-12-16T20:50:51.987971+0000 (oldest deadline 2023-12-16T20:51:16.687897+0000) 2023-12-16T21:01:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:29.665+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:30.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:30.301+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:30.103+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:30.651+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:31.267+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:31.126+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:31.643+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:32.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:32.302+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:32.084+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:33.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:01:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:01:32.582+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 7854 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:01:33.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:32.692+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:33.125+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:33.268+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:33.701+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:34.159+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:34.312+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:35.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:34.679+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:35.173+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:35.283+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:36.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:35.679+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:36.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:36.329+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:36.202+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:37.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:36.671+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:37.159+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:37.344+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:38.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:01:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:01:37.583+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 7947 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:01:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:37.704+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:38.279 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:38.126+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:38.701 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:38.296+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:39.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:38.697+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:39.169+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:39.309+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:39.674+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:40.190+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:40.358+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:40.669+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:41.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:41.353+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:41.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:41.190+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:41.875 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:41.623+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:42.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:42.225+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:42.345+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:43.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:01:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:01:42.584+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 8043 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:01:43.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:42.575+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:43.245+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:43.382+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:44.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:43.565+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:44.600 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:44.357+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:44.600 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:44.290+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:44.600 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:44.290+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:44.600 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:44.290+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:44.600 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:44.290+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:44.599+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:45.603 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:45.331+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:45.603 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:45.260+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:45.603 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:45.260+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:45.603 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:45.260+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:45.603 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:45.260+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:45.602+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:46.289+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:46.217+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:46.217+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:46.217+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:46.217+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:46.571+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:47.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:47.320+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:47.205+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:47.205+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:47.205+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:47.205+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:48.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:01:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:01:47.586+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 8135 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:01:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:47.581+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:48.276 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:48.229+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:48.277 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:48.229+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:48.277 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:48.229+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:48.277 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:48.229+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:48.574 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:48.275+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:48.573+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:49.623 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:49.265+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:49.623 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:49.265+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:49.623 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:49.265+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:49.623 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:49.265+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:49.623 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:49.265+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:49.623 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:49.265+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:49.623 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:49.265+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:49.623 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:49.265+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:50.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:49.622+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:50.240+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:50.240+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:50.240+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:50.241+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:50.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:50.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:50.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:50.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:50.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:50.236+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:50.616+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:51.245+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:51.245+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:51.245+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:51.245+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:51.253+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:51.253+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:51.253+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:51.253+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:52.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:51.588+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:52.248+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:52.248+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:52.248+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:52.248+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:52.291+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:52.291+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:52.291+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:52.291+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:53.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:01:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:01:52.587+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 8231 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:01:53.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:52.619+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:53.258 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:53.245+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:53.258 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:53.245+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:53.258 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:53.245+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:53.258 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:53.245+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:53.558 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:53.257+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:53.558 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:53.257+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:53.558 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:53.257+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:53.558 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:53.257+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:53.661+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:53.661+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:53.661+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:53.662+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:54.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:54.284+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:54.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:54.284+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:54.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:54.284+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:54.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:54.285+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:54.228+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:54.228+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:54.228+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:54.228+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:54.657+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:54.657+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:54.657+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:54.658+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:55.664 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:55.278+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:55.664 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:55.278+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:55.664 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:55.278+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:55.664 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:55.279+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:55.664 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:55.291+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:55.664 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:55.291+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:55.664 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:55.291+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:55.664 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:55.292+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:55.663+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:55.663+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:55.663+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:55.664+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:56.630 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:56.271+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:56.630 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:56.271+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:56.630 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:56.271+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:56.631 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:56.271+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:56.631 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:56.278+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:56.631 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:56.278+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:56.631 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:56.278+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:56.631 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:56.278+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:56.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:56.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:56.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:56.630+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:57.282+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:57.282+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:57.282+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:57.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:57.283+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:57.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:57.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:57.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:57.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:57.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:57.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:57.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:57.237+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:58.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:01:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:01:57.590+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 8325 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:01:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:57.582+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:57.582+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:57.582+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:57.583+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:58.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:58.257+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:58.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:58.257+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:58.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:58.257+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:58.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:58.257+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:58.190+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:58.190+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:58.190+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:58.191+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:58.598+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:58.598+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:58.598+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:01:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:58.599+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:59.233+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:59.233+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:59.233+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:01:59.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:01:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:01:59.234+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:01:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:59.261+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:59.261+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:59.261+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:01:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:01:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:01:59.261+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:59.601+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:59.601+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:59.601+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:01:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:01:59.601+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:00.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:00.184+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:00.184+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:00.184+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:00.185+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:00.293+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:00.293+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:00.293+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:00.294+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:00.621+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:00.621+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:00.621+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:00.622+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:01.278+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:01.278+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:01.278+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:01.278+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:01.230+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:01.230+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:01.230+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:01.231+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:01.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:01.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:01.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:01.572+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:02.244+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:02.244+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:02.244+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:02.244+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:02.220+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:02.220+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:02.220+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:02.220+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:03.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:02:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:02:02.591+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 8422 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:02:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:02.543+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:02.543+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:02.543+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:02.543+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:03.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:03.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:03.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:03.236+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:03.237+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:03.207+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:03.207+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:03.207+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:03.207+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:03.549+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:03.549+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:03.549+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:03.549+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:04.197+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:04.197+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:04.197+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:04.197+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:04.184+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:04.184+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:04.184+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:04.184+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:04.543+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:04.543+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:04.543+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:04.544+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:05.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:05.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:05.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:05.199+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:05.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:05.233+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:05.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:05.233+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:05.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:05.233+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:05.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:05.234+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:06.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:05.515+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:05.515+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:05.515+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:05.516+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:06.155+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:06.155+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:06.155+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:06.156+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:06.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:06.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:06.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:06.209+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:06.559+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:06.559+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:06.559+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:06.559+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:07.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:07.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:07.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:07.181+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:07.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:07.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:07.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:07.216+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:08.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:02:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:02:07.593+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 8515 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:02:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:07.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:07.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:07.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:07.585+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:08.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:08.154+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:08.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:08.154+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:08.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:08.154+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:08.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:08.154+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:08.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:08.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:08.173+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:08.173+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:08.574+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:08.574+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:08.574+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:08.574+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:09.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:09.176+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:09.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:09.176+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:09.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:09.176+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:09.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:09.176+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:09.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:09.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:09.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:09.179+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:09.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:09.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:09.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:09.590+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:10.140+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:10.140+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:10.140+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:10.141+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:10.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:10.149+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:10.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:10.149+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:10.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:10.149+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:10.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:10.149+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:10.628+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:10.628+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:10.628+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:10.629+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:11.121+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:11.121+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:11.121+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:11.122+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:11.157+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:11.157+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:11.157+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:11.157+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:11.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:11.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:11.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:11.666+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:12.118+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:12.118+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:12.118+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:12.119+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:12.139+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:12.139+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:12.139+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:12.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:12.139+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:13.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:02:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:02:12.594+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 8610 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:02:13.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:12.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:13.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:12.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:13.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:12.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:13.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:12.619+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:13.115+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:13.115+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:13.115+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:13.116+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:13.181+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:13.181+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:13.181+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:13.181+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:13.619+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:13.619+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:13.619+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:13.619+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:14.149+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:14.149+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:14.149+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:14.150+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:14.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:14.214+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:14.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:14.214+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:14.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:14.214+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:14.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:14.214+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:15.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:14.589+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:14.589+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:14.589+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:14.589+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:15.100+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:15.100+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:15.100+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:15.101+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:15.261+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:15.261+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:15.261+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:15.261+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:15.625+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:15.625+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:15.625+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:15.625+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:16.302+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:16.302+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:16.302+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:16.302+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:16.122+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:16.122+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:16.122+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:16.122+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:16.577+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:16.577+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:16.577+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:16.577+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:17.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:17.105+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:17.105+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:17.105+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:17.105+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:17.258+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:17.258+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:17.258+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:17.258+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:18.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:02:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:02:17.596+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 8704 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:02:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:17.582+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:17.582+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:17.582+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:17.582+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:18.272 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:18.127+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:18.272 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:18.127+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:18.272 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:18.127+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:18.272 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:18.128+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:18.572 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:18.271+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:18.572 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:18.271+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:18.572 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:18.271+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:18.572 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:18.272+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:19.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:18.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:18.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:18.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:18.572+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:19.169+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:19.169+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:19.169+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:19.170+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:19.260+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:19.260+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:19.260+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:19.260+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:20.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:19.602+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:20.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:19.602+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:20.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:19.602+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:19.602+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:20.161+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:20.161+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:20.161+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:20.162+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:20.295+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:20.295+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:20.295+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:20.295+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:21.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:20.641+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:20.641+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:20.641+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:20.641+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:21.115+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:21.115+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:21.115+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:21.115+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:21.254+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:21.254+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:21.254+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:21.254+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:21.607+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:21.607+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:21.607+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:21.607+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:22.107+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:22.107+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:22.107+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:22.108+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:22.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:22.223+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:22.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:22.223+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:22.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:22.223+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:22.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:22.223+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:23.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:02:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:02:22.598+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 8801 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:02:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:22.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:22.619+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:22.619+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:22.619+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:23.183+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:23.183+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:23.183+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:23.183+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:23.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:23.107+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:23.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:23.107+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:23.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:23.107+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:23.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:23.107+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:23.616+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:23.616+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:23.616+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:23.616+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:24.152+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:24.152+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:24.152+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:24.152+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:24.152+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:24.152+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:24.152+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:24.153+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:24.636+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:24.636+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:24.636+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:24.636+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:25.172+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:25.172+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:25.172+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:25.172+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:25.167+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:25.167+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:25.167+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:25.167+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:25.682+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:25.682+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:25.682+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:25.682+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:26.174+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:26.174+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:26.174+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:26.174+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:26.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:26.175+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:26.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:26.175+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:26.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:26.175+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:26.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:26.176+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:26.654+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:26.654+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:26.654+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:26.654+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:27.174+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:27.174+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:27.174+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:27.175+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:27.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:27.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:27.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:27.151+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:27.956 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:02:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:02:27.599+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 8893 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:02:27.956 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:27.703+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:27.956 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:27.703+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:27.956 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:27.703+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:27.957 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:27.703+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:28.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:28.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:28.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:28.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:28.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:28.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:28.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:28.159+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:28.181+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:28.181+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:28.181+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:28.181+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:29.142 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:28.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:29.142 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:28.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:29.143 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:28.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:29.143 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:28.752+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:29.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:29.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:29.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:29.199+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:29.142+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:29.142+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:29.142+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:29.142+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:30.112 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:29.766+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:30.112 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:29.766+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:30.112 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:29.766+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:30.112 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:29.766+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:30.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:30.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:30.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:30.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:30.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:30.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:30.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:30.209+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:30.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:30.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:30.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:30.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:30.112+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:31.028 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:30.757+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:31.028 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:30.757+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:31.028 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:30.757+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:31.028 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:30.757+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:31.208+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:31.208+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:31.208+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:31.209+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:31.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:31.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:31.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:31.120+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:31.745+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:31.745+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:31.745+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:31.746+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:32.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:32.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:32.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:32.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:32.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:32.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:32.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:32.217+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:32.106+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:32.106+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:32.106+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:32.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:32.106+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:33.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:02:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:02:32.601+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 8989 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:02:33.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:32.774+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:33.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:32.774+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:33.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:32.774+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:33.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:32.774+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:33.215+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:33.215+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:33.215+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:33.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:33.216+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:33.136+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:33.136+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:33.136+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:33.136+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:33.803+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:33.803+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:33.803+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:33.803+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:34.183+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:34.183+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:34.183+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:34.183+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:34.135+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:34.135+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:34.135+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:34.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:34.135+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:35.138 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:34.769+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:35.138 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:34.769+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:35.138 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:34.769+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:35.138 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:34.769+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:35.161+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:35.161+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:35.161+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:35.161+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:35.138+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:35.138+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:35.138+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:35.138+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:36.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:35.737+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:35.737+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:35.737+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:35.737+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:36.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:36.120+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:36.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:36.120+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:36.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:36.120+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:36.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:36.120+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:36.137+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:36.137+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:36.137+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:36.138+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:37.083 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:36.782+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:37.084 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:36.782+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:37.084 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:36.782+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:37.084 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:36.782+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:37.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:37.171+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:37.171+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:37.171+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:37.171+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:37.083+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:37.083+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:37.083+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:37.084+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:38.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:02:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:02:37.603+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 9083 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:02:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:37.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:37.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:37.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:37.797+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:38.133+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:38.133+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:38.133+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:38.133+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:38.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:38.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:38.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:38.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:38.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:38.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:38.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:38.209+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:39.118 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:38.792+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:39.118 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:38.792+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:39.118 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:38.792+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:39.118 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:38.792+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:39.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:39.249+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:39.249+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:39.249+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:39.249+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:39.117+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:39.117+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:39.117+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:39.118+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:40.107 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:39.786+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:40.107 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:39.786+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:40.108 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:39.786+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:40.108 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:39.786+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:40.107+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:40.107+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:40.107+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:40.108+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:40.234+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:40.234+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:40.234+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:40.234+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:41.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:41.064+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:41.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:41.064+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:41.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:41.064+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:41.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:41.064+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:41.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:40.816+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:41.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:40.816+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:41.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:40.816+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:41.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:40.816+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:41.669 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:41.274+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:41.669 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:41.274+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:41.669 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:41.274+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:41.669 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:41.274+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:41.795+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:41.795+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:41.795+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:41.796+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:42.102+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:42.102+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:42.102+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:42.102+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:42.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:42.263+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:42.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:42.263+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:42.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:42.263+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:42.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:42.263+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:43.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:02:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:02:42.604+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 9180 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:02:43.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:42.788+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:43.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:42.788+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:43.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:42.789+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:43.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:42.789+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:43.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:43.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:43.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:43.307+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:43.104+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:43.104+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:43.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:43.104+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:43.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:43.104+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:43.750+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:43.750+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:43.750+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:43.750+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:44.150+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:44.150+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:44.150+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:44.150+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:44.326+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:44.326+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:44.326+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:44.326+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:45.177 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:44.789+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:45.178 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:44.789+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:45.178 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:44.789+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:45.178 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:44.789+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:45.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:45.326+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:45.326+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:45.326+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:45.326+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:45.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:45.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:45.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:45.178+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:45.745+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:45.745+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:45.745+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:45.745+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:46.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:46.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:46.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:46.179+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:46.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:46.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:46.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:46.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:46.374+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:47.150 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:46.768+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:47.150 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:46.768+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:47.150 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:46.768+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:47.150 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:46.768+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:47.424 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:47.150+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:47.424 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:47.150+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:47.425 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:47.150+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:47.425 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:47.150+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:47.731 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:02:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:02:47.606+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 9273 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:02:47.731 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:47.424+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:47.732 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:47.424+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:47.732 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:47.424+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:47.732 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:47.424+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:48.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:47.762+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:47.762+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:47.762+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:47.762+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:48.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:48.114+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:48.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:48.114+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:48.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:48.114+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:48.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:48.115+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:48.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:48.417+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:48.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:48.417+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:48.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:48.417+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:48.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:48.417+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:49.117 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:48.785+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:49.117 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:48.785+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:49.117 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:48.785+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:49.117 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:48.785+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:49.381 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:49.117+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:49.381 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:49.117+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:49.382 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:49.117+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:49.382 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:49.117+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:49.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:49.381+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:49.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:49.381+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:49.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:49.381+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:49.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:49.381+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:50.119 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:49.819+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:50.120 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:49.819+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:50.120 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:49.819+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:50.120 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:49.819+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:50.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:50.120+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:50.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:50.120+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:50.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:50.120+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:50.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:50.120+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:50.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:50.417+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:50.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:50.417+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:50.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:50.417+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:50.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:50.417+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:51.148 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:50.828+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:51.148 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:50.828+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:51.148 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:50.828+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:51.148 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:50.829+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:51.451 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:51.148+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:51.451 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:51.148+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:51.451 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:51.148+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:51.451 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:51.148+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:51.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:51.451+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:51.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:51.451+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:51.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:51.451+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:51.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:51.451+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:52.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:51.866+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:52.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:51.866+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:52.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:51.866+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:52.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:51.866+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:52.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:52.104+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:52.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:52.104+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:52.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:52.104+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:52.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:52.105+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:52.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:02:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:02:52.607+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 9369 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:02:52.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:52.498+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:52.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:52.498+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:52.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:52.498+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:52.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:52.498+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:53.095+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:53.095+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:53.095+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:53.095+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:53.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:52.880+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:53.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:52.880+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:53.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:52.880+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:53.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:52.880+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:53.925 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:53.514+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:53.925 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:53.514+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:53.925 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:53.514+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:53.925 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:53.514+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:54.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:54.103+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:54.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:54.103+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:54.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:54.103+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:54.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:54.104+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:54.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:53.925+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:54.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:53.925+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:54.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:53.925+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:54.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:53.925+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:54.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:54.490+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:54.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:54.490+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:54.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:54.490+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:54.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:54.490+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:55.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:55.061+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:55.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:55.061+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:55.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:55.061+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:55.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:55.061+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:55.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:54.890+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:55.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:54.890+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:55.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:54.890+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:55.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:54.890+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:55.863 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:55.519+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:55.863 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:55.519+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:55.863 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:55.519+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:55.863 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:55.519+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:56.017+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:56.017+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:56.017+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:56.017+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:56.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:55.863+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:56.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:55.863+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:56.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:55.863+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:56.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:55.863+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:56.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:56.469+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:56.469+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:56.469+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:56.470+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:57.061+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:57.061+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:57.061+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:57.061+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:57.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:56.837+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:57.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:56.837+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:57.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:56.837+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:57.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:56.837+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:57.735 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:02:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:02:57.609+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 9463 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:02:57.736 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:57.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:57.736 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:57.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:57.736 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:57.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:57.736 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:57.448+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:57.801+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:57.801+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:57.801+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:58.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:57.801+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:58.435 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:58.044+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:58.435 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:58.044+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:58.436 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:58.044+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:58.436 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:58.044+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:58.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:58.435+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:58.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:58.435+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:58.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:58.435+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:58.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:58.435+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:59.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:58.995+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:59.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:58.995+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:59.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:58.995+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:02:59.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:02:58.995+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:59.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:58.809+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:59.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:58.809+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:59.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:58.809+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:02:59.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:58.809+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:02:59.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:59.463+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:59.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:59.463+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:59.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:59.463+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:02:59.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:02:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:02:59.463+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:00.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:59.827+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:00.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:59.828+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:00.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:59.828+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:00.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:02:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:02:59.828+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:00.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:00.042+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:00.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:00.042+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:00.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:00.042+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:00.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:00.043+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:00.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:00.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:00.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:00.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:00.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:00.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:00.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:00.416+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:01.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:00.850+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:01.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:00.850+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:01.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:00.850+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:01.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:00.850+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:01.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:01.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:01.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:01.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:01.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:01.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:01.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:01.082+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:01.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:01.455+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:01.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:01.455+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:01.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:01.455+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:01.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:01.455+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:02.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:02.062+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:02.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:02.062+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:02.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:02.062+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:02.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:02.062+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:02.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:01.826+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:02.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:01.826+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:02.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:01.826+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:02.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:01.826+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:02.736 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:03:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:03:02.610+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 9559 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:03:02.736 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:02.440+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:02.736 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:02.440+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:02.736 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:02.440+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:02.736 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:02.440+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:03.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:02.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:02.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:02.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:03.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:02.777+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:03.409 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:03.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:03.410 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:03.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:03.410 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:03.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:03.410 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:03.039+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:03.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:03.409+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:03.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:03.409+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:03.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:03.409+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:03.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:03.409+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:03.991+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:03.991+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:03.991+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:03.991+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:03.780+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:03.780+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:03.780+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:03.780+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:04.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:04.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:04.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:04.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:04.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:04.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:04.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:04.408+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:05.031+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:05.031+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:05.031+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:05.032+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:04.812+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:04.812+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:04.812+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:05.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:04.812+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:05.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:05.424+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:05.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:05.424+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:05.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:05.424+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:05.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:05.424+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:06.031 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:05.765+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:06.032 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:05.765+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:06.032 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:05.765+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:06.032 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:05.765+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:06.388 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:06.031+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:06.388 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:06.031+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:06.388 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:06.031+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:06.388 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:06.031+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:06.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:06.387+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:06.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:06.387+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:06.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:06.387+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:06.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:06.387+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:07.054 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:06.763+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:07.054 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:06.763+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:07.054 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:06.763+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:07.054 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:06.764+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:07.367 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:07.054+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:07.367 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:07.054+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:07.368 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:07.054+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:07.368 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:07.054+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:07.625 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:03:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:03:07.612+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 9653 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:03:07.625 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:07.366+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:07.625 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:07.366+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:07.625 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:07.366+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:07.625 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:07.366+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:07.892 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:07.768+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:07.892 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:07.768+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:07.892 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:07.768+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:07.892 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:07.769+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:08.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:08.092+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:08.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:08.092+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:08.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:08.092+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:08.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:08.092+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:08.728 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:08.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:08.728 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:08.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:08.729 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:08.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:08.729 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:08.398+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:09.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:08.728+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:08.728+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:08.728+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:08.728+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:09.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:09.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:09.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:09.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:09.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:09.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:09.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:09.139+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:09.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:09.418+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:09.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:09.418+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:09.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:09.418+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:09.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:09.418+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:10.094 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:09.775+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:10.094 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:09.775+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:10.094 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:09.775+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:10.094 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:09.775+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:10.443 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:10.093+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:10.443 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:10.093+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:10.443 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:10.093+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:10.443 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:10.093+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:10.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:10.442+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:10.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:10.442+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:10.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:10.442+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:10.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:10.442+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:11.101 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:10.793+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:11.101 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:10.793+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:11.101 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:10.793+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:11.101 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:10.793+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:11.479 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:11.100+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:11.479 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:11.100+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:11.480 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:11.100+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:11.480 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:11.100+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:11.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:11.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:11.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:11.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:11.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:11.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:11.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:11.479+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:12.145 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:11.781+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:12.145 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:11.781+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:12.145 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:11.781+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:12.145 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:11.781+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:12.474 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:12.144+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:12.474 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:12.144+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:12.474 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:12.144+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:12.474 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:12.144+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:12.740 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:03:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:03:12.613+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 9748 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:03:12.740 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:12.473+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:12.740 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:12.473+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:12.741 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:12.473+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:12.741 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:12.473+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:13.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:12.768+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:13.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:12.768+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:13.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:12.768+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:13.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:12.768+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:13.121+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:13.121+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:13.121+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:13.121+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:14.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:13.513+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:14.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:13.513+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:14.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:13.513+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:14.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:13.514+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:13.734+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:13.734+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:13.734+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:13.734+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:14.161+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:14.161+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:14.161+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:14.161+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:15.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:14.525+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:15.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:14.525+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:15.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:14.525+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:15.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:14.525+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:14.689+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:14.689+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:14.689+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:14.689+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:15.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:15.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:15.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:15.210+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:15.532+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:15.532+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:15.532+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:15.532+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:15.703+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:15.703+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:15.703+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:15.703+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:16.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:16.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:16.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:16.179+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:16.504+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:16.504+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:16.504+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:16.504+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:16.666+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:16.666+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:16.666+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:17.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:16.666+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:17.169+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:17.169+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:17.169+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:17.170+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:18.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:03:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:03:17.616+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 9843 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:03:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:17.716+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:17.716+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:17.716+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:17.717+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:18.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:17.538+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:18.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:17.538+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:18.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:17.538+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:18.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:17.538+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:18.159+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:18.159+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:18.159+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:18.160+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:19.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:18.587+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:18.587+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:18.587+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:18.587+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:18.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:18.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:18.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:18.710+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:19.183+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:19.183+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:19.183+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:19.183+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:20.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:19.635+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:20.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:19.635+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:20.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:19.635+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:20.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:19.635+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:19.708+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:19.708+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:19.708+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:19.709+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:20.171+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:20.171+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:20.171+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:20.172+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:20.604+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:20.604+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:20.604+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:20.604+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:20.696+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:21.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:20.696+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:21.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:20.696+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:21.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:20.696+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:21.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:21.147+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:21.147+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:21.147+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:21.148+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:21.571+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:21.571+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:21.571+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:21.571+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:21.673+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:21.673+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:21.673+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:21.674+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:22.133+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:22.133+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:22.133+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:22.134+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:23.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:03:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:03:22.618+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 9936 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:03:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:22.639+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:22.639+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:22.639+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:22.639+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:23.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:22.549+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:23.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:22.549+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:23.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:22.549+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:23.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:22.550+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:23.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:23.134+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:23.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:23.134+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:23.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:23.134+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:23.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:23.135+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:24.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:23.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:23.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:23.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:23.607+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:24.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:23.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:24.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:23.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:24.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:23.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:24.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:23.563+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:24.173+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:24.173+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:24.173+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:24.173+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:25.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:24.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:24.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:24.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:24.586+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:24.531+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:24.531+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:24.531+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:24.531+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:25.160+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:25.160+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:25.160+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:25.160+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:26.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:25.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:25.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:25.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:25.631+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:25.540+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:25.540+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:25.540+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:25.540+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:26.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:26.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:26.158+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:26.159+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:27.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:26.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:26.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:26.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:26.607+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:27.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:26.560+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:27.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:26.560+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:27.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:26.560+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:27.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:26.560+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:27.194+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:27.194+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:27.194+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:27.194+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:28.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:03:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:03:27.619+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 10024 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:03:28.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:27.557+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:28.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:27.557+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:28.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:27.557+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:28.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:27.558+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:28.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:27.586+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:28.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:27.586+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:28.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:27.586+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:28.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:27.586+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:28.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:28.192+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:28.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:28.192+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:28.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:28.192+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:28.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:28.192+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:29.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:28.562+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:28.562+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:28.562+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:28.563+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:29.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:28.598+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:29.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:28.598+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:29.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:28.598+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:29.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:28.598+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:29.242+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:29.242+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:29.242+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:29.242+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:29.542+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:29.542+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:29.542+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:29.543+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:30.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:29.623+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:30.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:29.623+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:30.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:29.623+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:30.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:29.624+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:30.586 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:30.258+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:30.586 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:30.258+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:30.586 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:30.258+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:30.586 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:30.258+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:31.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:30.588+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:30.588+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:30.588+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:30.588+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:31.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:30.586+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:31.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:30.586+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:31.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:30.586+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:31.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:30.586+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:31.561 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:31.259+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:31.561 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:31.259+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:31.561 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:31.259+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:31.561 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:31.260+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:31.560+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:31.560+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:31.560+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:31.561+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:32.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:31.615+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:32.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:31.615+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:32.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:31.615+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:32.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:31.615+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:32.571 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:32.303+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:32.571 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:32.303+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:32.571 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:32.303+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:32.571 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:32.303+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:33.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:03:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:03:32.621+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 10109 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:03:33.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:32.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:33.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:32.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:33.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:32.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:33.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:32.571+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:33.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:32.634+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:33.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:32.634+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:33.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:32.634+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:33.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:32.634+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:33.564 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:33.256+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:33.564 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:33.256+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:33.565 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:33.256+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:33.565 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:33.257+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:33.602+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:33.602+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:33.602+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:33.602+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:34.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:33.683+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:34.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:33.683+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:34.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:33.683+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:34.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:33.683+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:34.619 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:34.275+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:34.619 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:34.275+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:34.619 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:34.275+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:34.619 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:34.276+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:34.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:34.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:34.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:34.619+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:34.713+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:34.713+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:34.713+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:34.714+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:35.539 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:35.255+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:35.539 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:35.255+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:35.539 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:35.255+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:35.540 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:35.255+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:36.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:35.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:35.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:35.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:35.606+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:35.759+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:35.759+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:35.759+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:35.759+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:36.636 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:36.277+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:36.636 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:36.277+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:36.636 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:36.277+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:36.636 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:36.277+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:36.636+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:36.636+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:36.636+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:36.636+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:37.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:36.735+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:37.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:36.735+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:37.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:36.735+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:37.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:36.736+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:37.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:37.237+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:37.237+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:37.237+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:37.237+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:38.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:03:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:03:37.623+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 10195 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:03:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:37.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:37.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:37.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:37.606+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:38.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:37.763+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:38.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:37.763+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:38.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:37.763+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:38.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:37.763+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:38.210+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:38.210+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:38.210+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:38.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:38.211+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:38.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:38.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:38.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:38.572+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:39.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:38.745+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:39.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:38.745+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:39.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:38.745+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:39.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:38.745+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:39.229+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:39.229+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:39.229+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:39.230+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:39.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:39.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:39.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:39.573+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:40.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:39.790+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:40.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:39.790+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:40.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:39.790+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:40.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:39.791+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:40.225+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:40.225+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:40.225+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:40.226+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:40.800+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:40.800+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:40.800+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:40.800+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:40.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:40.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:40.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:40.613+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:41.249+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:41.249+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:41.249+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:41.249+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:42.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:41.834+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:42.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:41.834+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:42.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:41.834+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:42.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:41.834+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:41.641+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:41.641+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:42.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:41.641+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:42.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:41.642+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:42.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:42.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:42.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:42.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:42.209+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:43.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:03:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:03:42.625+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 10282 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:03:43.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:42.655+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:43.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:42.655+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:43.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:42.655+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:43.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:42.655+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:43.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:42.834+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:43.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:42.834+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:43.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:42.834+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:43.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:42.834+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:43.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:43.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:43.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:43.180+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:43.181+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:43.654+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:43.654+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:43.654+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:43.654+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:44.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:43.796+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:44.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:43.796+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:44.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:43.796+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:44.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:43.796+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:44.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:44.153+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:44.153+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:44.153+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:44.153+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:44.664+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:44.664+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:44.664+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:44.665+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:45.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:44.764+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:45.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:44.764+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:45.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:44.764+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:45.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:44.764+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:45.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:45.174+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:45.174+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:45.174+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:45.174+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:46.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:45.677+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:45.677+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:45.677+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:45.677+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:46.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:45.770+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:46.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:45.770+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:46.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:45.770+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:46.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:45.771+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:46.177+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:46.177+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:46.177+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:46.177+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:46.714+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:46.714+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:46.714+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:46.715+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:47.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:46.789+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:47.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:46.789+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:47.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:46.789+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:47.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:46.789+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:47.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:47.218+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:47.218+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:47.218+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:48.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:03:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:03:47.627+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 10369 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:03:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:47.753+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:47.753+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:47.753+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:47.754+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:48.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:47.809+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:48.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:47.809+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:48.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:47.809+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:48.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:47.809+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:48.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:48.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:48.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:48.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:48.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:48.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:48.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:48.210+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:49.185 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:48.852+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:49.185 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:48.852+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:49.185 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:48.852+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:49.185 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:48.852+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:49.185 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:48.787+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:49.185 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:48.787+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:49.185 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:48.787+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:49.185 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:48.787+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:49.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:49.185+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:49.185+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:49.185+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:49.185+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:50.146 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:49.775+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:50.147 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:49.775+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:50.147 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:49.775+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:50.147 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:49.776+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:50.147 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:49.846+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:50.147 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:49.846+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:50.147 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:49.846+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:50.147 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:49.846+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:50.146+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:50.146+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:50.146+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:50.147+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:51.175 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:50.879+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:51.175 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:50.879+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:51.175 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:50.879+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:51.175 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:50.879+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:51.176 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:50.795+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:51.176 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:50.795+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:51.176 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:50.795+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:51.176 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:50.795+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:51.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:51.175+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:51.175+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:51.175+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:51.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:51.175+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:51.815+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:51.815+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:51.815+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:51.816+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:51.883+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:51.883+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:51.883+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:51.883+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:52.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:52.163+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:52.163+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:52.163+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:52.163+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:52.903 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:03:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:03:52.628+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 10459 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:03:52.903 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:52.848+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:52.903 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:52.848+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:52.903 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:52.848+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:52.903 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:52.849+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:53.179 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:52.902+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:53.179 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:52.902+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:53.180 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:52.902+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:53.180 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:52.903+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:53.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:53.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:53.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:53.180+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:54.202 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:53.830+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:54.202 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:53.830+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:54.202 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:53.830+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:54.202 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:53.830+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:54.202 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:53.887+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:54.202 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:53.887+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:54.202 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:53.887+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:54.202 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:53.887+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:54.202+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:54.202+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:54.202+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:54.202+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:55.241 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:54.835+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:55.241 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:54.835+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:55.241 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:54.835+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:55.241 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:54.835+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:55.242 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:54.909+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:55.242 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:54.909+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:55.242 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:54.909+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:55.242 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:54.909+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:55.240+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:55.241+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:55.241+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:55.241+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:56.246 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:55.879+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:56.246 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:55.879+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:56.246 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:55.879+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:56.246 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:55.879+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:56.246 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:55.877+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:56.246 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:55.877+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:56.246 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:55.877+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:56.246 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:55.878+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:56.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:56.246+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:56.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:56.246+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:56.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:56.246+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:56.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:56.246+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:57.200 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:56.899+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:57.200 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:56.899+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:57.200 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:56.899+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:57.200 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:56.899+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:57.200 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:56.899+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:57.201 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:56.899+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:57.201 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:56.899+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:57.201 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:56.899+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:57.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:57.199+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:57.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:57.199+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:57.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:57.199+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:57.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:57.200+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:57.896 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:03:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:03:57.630+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 10553 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:03:58.154 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:57.896+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:58.154 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:57.896+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:58.155 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:57.896+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:58.155 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:57.896+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:58.155 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:57.906+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:58.155 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:57.906+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:58.155 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:57.906+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:58.155 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:57.906+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:58.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:58.154+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:58.154+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:58.154+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:58.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:58.154+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:59.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:58.935+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:59.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:58.935+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:59.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:58.935+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:03:59.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:58.936+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:59.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:58.932+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:59.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:58.932+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:59.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:58.932+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:03:59.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:58.932+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:03:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:59.172+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:59.173+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:59.173+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:03:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:03:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:03:59.173+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:00.210 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:59.915+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:00.210 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:59.915+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:00.210 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:59.915+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:00.210 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:03:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:03:59.916+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:00.210 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:59.898+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:00.210 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:59.898+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:00.210 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:59.898+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:00.210 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:03:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:03:59.898+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:00.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:00.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:00.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:00.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:00.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:00.209+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:00.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:00.210+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:01.200 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:00.923+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:01.200 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:00.923+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:01.200 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:00.923+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:01.200 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:00.923+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:01.200 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:00.923+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:01.200 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:00.923+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:01.200 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:00.923+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:01.200 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:00.923+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:01.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:01.199+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:01.199+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:01.199+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:01.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:01.200+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:02.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:01.922+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:02.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:01.922+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:02.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:01.922+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:02.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:01.922+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:01.921+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:01.921+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:01.921+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:01.921+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:02.164+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:02.164+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:02.164+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:02.164+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:02.892 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:04:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:04:02.632+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 10647 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:04:03.146 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:02.892+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:03.146 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:02.892+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:03.146 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:02.892+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:03.146 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:02.892+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:03.147 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:02.962+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:03.147 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:02.962+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:03.147 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:02.962+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:03.147 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:02.963+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:03.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:03.145+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:03.145+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:03.145+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:03.146+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:04.139 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:03.855+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:04.139 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:03.855+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:04.139 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:03.855+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:04.139 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:03.855+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:04.140 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:03.914+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:04.140 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:03.914+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:04.140 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:03.914+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:04.140 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:03.914+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:04.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:04.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:04.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:04.139+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:05.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:04.858+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:04.858+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:04.858+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:04.858+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:05.097+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:05.097+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:05.097+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:05.097+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:05.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:04.960+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:05.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:04.960+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:05.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:04.960+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:05.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:04.960+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:06.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:05.864+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:06.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:05.864+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:06.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:05.864+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:06.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:05.864+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:06.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:06.109+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:06.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:06.109+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:06.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:06.109+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:06.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:06.109+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:06.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:05.932+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:06.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:05.932+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:06.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:05.932+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:06.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:05.932+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:07.154 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:06.818+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:07.154 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:06.818+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:07.155 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:06.818+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:07.155 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:06.819+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:07.155 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:06.924+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:07.155 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:06.924+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:07.155 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:06.924+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:07.156 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:06.924+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:07.154+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:07.154+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:07.154+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:07.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:07.154+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:08.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:04:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:04:07.633+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 10741 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:04:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:07.814+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:08.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:07.814+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:08.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:07.814+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:08.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:07.814+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:08.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:07.879+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:08.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:07.879+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:08.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:07.879+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:08.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:07.879+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:08.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:08.185+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:08.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:08.185+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:08.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:08.185+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:08.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:08.185+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:09.143 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:08.780+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:09.143 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:08.780+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:09.143 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:08.780+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:09.143 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:08.780+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:09.143 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:08.884+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:09.143 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:08.884+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:09.144 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:08.884+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:09.144 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:08.885+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:09.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:09.143+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:09.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:09.143+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:09.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:09.143+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:09.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:09.143+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:10.150 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:09.780+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:10.151 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:09.780+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:10.151 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:09.780+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:10.151 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:09.780+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:10.151 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:09.862+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:10.151 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:09.862+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:10.151 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:09.862+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:10.151 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:09.862+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:10.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:10.150+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:10.150+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:10.150+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:10.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:10.151+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:11.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:10.740+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:10.740+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:10.740+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:10.740+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:11.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:10.835+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:11.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:10.835+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:11.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:10.835+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:11.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:10.835+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:11.185+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:11.185+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:11.185+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:11.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:11.185+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:11.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:11.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:11.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:11.727+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:11.786+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:11.786+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:11.786+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:11.786+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:12.196+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:12.196+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:12.196+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:12.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:12.196+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:13.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:04:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:04:12.635+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 10836 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:04:13.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:12.682+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:13.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:12.682+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:13.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:12.682+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:13.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:12.682+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:13.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:12.784+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:13.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:12.784+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:13.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:12.784+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:13.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:12.784+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:13.240+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:13.240+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:13.240+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:13.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:13.240+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:14.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:13.691+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:13.691+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:13.691+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:13.692+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:14.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:13.767+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:14.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:13.767+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:14.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:13.767+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:14.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:13.767+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:14.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:14.213+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:14.213+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:14.213+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:14.213+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:14.654+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:14.654+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:14.654+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:14.654+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:15.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:14.777+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:15.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:14.777+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:15.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:14.777+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:15.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:14.778+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:15.475 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:15.190+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:15.475 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:15.190+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:15.475 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:15.190+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:15.476 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:15.190+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:15.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:15.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:15.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:15.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:15.612+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:16.211 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:15.788+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:16.211 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:15.788+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:16.211 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:15.788+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:16.211 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:15.788+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:16.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:16.210+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:16.210+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:16.210+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:16.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:16.211+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:16.656+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:16.656+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:16.656+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:16.657+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:16.765+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:16.765+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:16.765+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:16.765+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:17.222+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:17.222+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:17.222+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:17.222+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:18.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:04:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:04:17.637+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 10929 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:04:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:17.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:17.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:17.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:18.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:17.611+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:18.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:17.769+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:18.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:17.769+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:18.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:17.769+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:18.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:17.769+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:18.207+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:18.207+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:18.207+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:18.208+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:18.566+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:18.566+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:18.566+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:18.566+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:18.810+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:18.810+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:18.810+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:18.811+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:19.232+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:19.232+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:19.232+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:19.232+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:19.778 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:19.525+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:19.778 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:19.525+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:19.779 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:19.525+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:19.779 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:19.525+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:20.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:19.778+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:20.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:19.778+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:20.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:19.778+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:20.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:19.778+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:20.228+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:20.228+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:20.228+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:20.228+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:21.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:20.573+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:20.573+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:20.573+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:20.573+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:20.736+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:20.736+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:20.736+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:20.736+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:21.556 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:21.270+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:21.556 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:21.270+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:21.556 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:21.270+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:21.556 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:21.271+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:22.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:21.555+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:21.555+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:21.555+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:21.555+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:21.691+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:21.691+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:21.691+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:21.691+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:22.605 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:22.321+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:22.606 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:22.321+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:22.606 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:22.321+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:22.606 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:22.321+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:23.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:04:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:04:22.638+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 11024 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:04:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:22.604+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:22.604+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:22.604+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:22.605+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:23.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:22.720+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:23.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:22.720+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:23.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:22.720+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:23.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:22.720+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:23.614 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:23.321+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:23.615 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:23.321+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:23.615 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:23.321+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:23.615 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:23.321+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:23.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:23.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:23.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:23.614+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:24.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:23.688+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:24.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:23.688+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:24.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:23.688+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:24.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:23.688+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:24.642 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:24.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:24.643 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:24.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:24.643 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:24.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:24.643 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:24.338+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:24.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:24.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:24.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:24.642+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:24.697+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:24.697+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:24.697+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:24.698+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:25.598 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:25.318+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:25.599 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:25.318+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:25.599 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:25.318+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:25.599 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:25.318+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:25.683+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:25.683+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:25.683+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:25.683+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:25.598+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:25.598+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:25.598+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:25.598+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:26.635 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:26.331+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:26.635 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:26.331+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:26.635 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:26.331+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:26.635 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:26.332+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:26.634+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:26.634+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:26.634+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:26.634+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:27.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:26.712+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:27.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:26.712+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:27.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:26.712+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:27.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:26.712+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:27.641 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:27.352+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:27.642 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:27.352+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:27.642 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:27.352+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:27.642 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:27.352+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:28.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:04:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:04:27.640+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 11117 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:04:28.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:27.680+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:28.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:27.680+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:28.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:27.680+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:28.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:27.681+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:28.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:27.656+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:28.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:27.656+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:28.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:27.656+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:28.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:27.656+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:28.663 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:28.342+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:28.663 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:28.342+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:28.663 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:28.342+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:28.664 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:28.342+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:28.663+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:28.663+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:28.663+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:28.664+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:29.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:28.663+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:29.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:28.663+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:29.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:28.663+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:29.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:28.663+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:29.648 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:29.316+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:29.648 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:29.316+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:29.648 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:29.316+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:29.648 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:29.317+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:30.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:29.655+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:29.655+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:29.655+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:29.656+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:30.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:29.646+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:30.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:29.646+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:30.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:29.646+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:30.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:29.647+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:30.610 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:30.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:30.610 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:30.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:30.610 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:30.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:30.610 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:30.337+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:31.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:30.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:30.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:30.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:30.610+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:31.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:30.690+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:31.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:30.690+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:31.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:30.690+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:31.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:30.690+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:31.678 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:31.345+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:31.678 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:31.345+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:31.678 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:31.345+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:31.678 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:31.345+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:31.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:31.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:31.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:31.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:31.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:31.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:31.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:31.586+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:32.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:31.720+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:32.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:31.720+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:32.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:31.720+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:32.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:31.720+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:32.643 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:32.563+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:32.643 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:32.563+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:32.643 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:32.563+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:32.644 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:32.563+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:32.644 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:32.336+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:32.644 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:32.336+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:32.644 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:32.336+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:32.644 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:32.336+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:33.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:04:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:04:32.642+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 11212 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:04:33.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:32.705+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:33.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:32.705+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:33.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:32.705+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:33.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:32.705+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:33.663 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:33.523+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:33.663 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:33.523+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:33.663 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:33.523+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:33.663 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:33.523+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:33.664 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:33.360+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:33.664 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:33.360+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:33.664 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:33.360+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:33.664 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:33.360+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:34.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:33.663+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:34.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:33.663+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:34.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:33.663+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:34.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:33.663+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:34.703 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:34.503+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:34.703 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:34.503+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:34.703 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:34.503+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:34.703 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:34.503+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:34.703 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:34.397+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:34.704 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:34.397+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:34.704 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:34.397+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:34.704 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:34.397+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:34.702+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:34.702+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:34.702+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:34.702+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:35.691 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:35.378+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:35.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:35.378+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:35.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:35.378+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:35.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:35.378+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:35.692 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:35.550+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:35.692 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:35.550+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:35.692 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:35.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:35.692 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:35.551+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:35.691+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:35.691+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:35.691+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:35.692+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:36.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:36.554+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:36.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:36.554+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:36.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:36.554+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:36.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:36.555+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:36.699 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:36.379+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:36.699 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:36.379+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:36.699 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:36.379+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:36.699 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:36.379+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:36.980 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:36.698+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:36.980 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:36.698+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:36.980 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:36.698+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:36.980 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:36.698+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:37.645 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:37.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:37.646 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:37.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:37.646 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:37.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:37.646 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:37.585+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:37.647 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:37.344+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:37.647 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:37.345+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:37.647 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:37.345+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:37.647 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:37.345+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:38.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:04:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:04:37.645+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 11307 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:04:38.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:37.660+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:38.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:37.660+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:38.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:37.660+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:38.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:37.660+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:38.591+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:38.591+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:38.591+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:38.591+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:38.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:38.616+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:38.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:38.616+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:38.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:38.616+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:38.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:38.616+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:38.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:38.389+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:38.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:38.389+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:38.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:38.389+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:38.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:38.390+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:39.545+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:39.545+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:39.545+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:39.545+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:39.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:39.591+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:39.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:39.591+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:39.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:39.591+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:39.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:39.591+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:39.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:39.369+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:39.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:39.369+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:39.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:39.369+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:39.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:39.370+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:40.392+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:40.392+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:40.392+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:40.393+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:40.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:40.613+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:40.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:40.613+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:40.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:40.613+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:40.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:40.613+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:40.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:40.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:40.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:40.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:40.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:40.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:40.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:40.558+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:41.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:41.437+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:41.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:41.437+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:41.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:41.437+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:41.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:41.437+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:41.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:41.521+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:41.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:41.521+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:41.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:41.521+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:41.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:41.522+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:41.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:41.600+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:41.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:41.600+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:41.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:41.600+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:41.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:41.601+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:42.657 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:04:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:04:42.646+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 11403 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:04:42.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:42.400+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:42.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:42.400+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:42.658 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:42.400+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:42.658 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:42.401+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:42.658 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:42.531+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:42.658 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:42.531+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:42.658 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:42.531+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:42.658 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:42.532+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:42.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:42.604+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:42.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:42.604+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:42.659 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:42.604+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:42.659 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:42.604+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:43.370+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:43.370+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:43.370+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:43.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:43.370+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:44.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:43.555+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:44.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:43.555+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:44.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:43.555+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:44.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:43.555+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:43.567+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:43.567+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:43.567+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:44.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:43.567+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:44.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:44.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:44.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:44.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:44.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:44.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:44.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:44.508+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:44.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:44.411+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:44.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:44.411+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:44.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:44.411+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:44.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:44.411+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:44.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:44.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:44.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:44.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:44.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:44.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:44.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:44.551+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:45.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:45.421+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:45.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:45.421+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:45.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:45.421+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:45.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:45.421+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:45.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:45.504+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:45.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:45.504+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:45.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:45.504+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:45.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:45.504+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:45.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:45.491+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:45.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:45.491+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:45.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:45.491+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:45.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:45.491+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:46.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:46.461+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:46.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:46.461+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:46.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:46.461+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:46.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:46.461+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:46.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:46.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:46.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:46.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:46.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:46.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:46.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:46.465+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:46.385+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:46.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:46.385+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:46.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:46.385+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:46.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:46.386+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:47.649 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:47.385+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:47.649 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:47.385+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:47.649 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:47.385+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:47.649 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:47.385+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:47.649 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:47.419+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:47.649 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:47.419+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:47.649 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:47.419+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:47.649 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:47.420+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:47.650 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:47.473+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:47.650 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:47.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:47.650 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:47.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:47.650 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:47.474+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:48.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:04:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:04:47.648+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 11496 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:04:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:48.410+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:48.410+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:48.410+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:48.410+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:48.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:48.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:48.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:48.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:48.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:48.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:48.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:48.432+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:48.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:48.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:48.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:48.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:48.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:48.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:48.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:48.430+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:49.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:49.446+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:49.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:49.446+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:49.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:49.446+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:49.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:49.446+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:49.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:49.399+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:49.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:49.399+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:49.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:49.399+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:49.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:49.399+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:49.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:49.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:49.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:49.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:49.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:49.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:49.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:49.463+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:50.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:50.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:50.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:50.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:50.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:50.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:50.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:50.400+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:50.421+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:50.421+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:50.421+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:50.422+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:50.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:50.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:50.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:50.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:50.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:50.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:50.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:50.439+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:51.679 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:51.363+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:51.679 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:51.363+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:51.679 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:51.363+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:51.679 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:51.363+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:51.679 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:51.392+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:51.679 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:51.392+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:51.680 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:51.392+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:51.680 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:51.392+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:51.680 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:51.426+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:51.680 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:51.426+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:51.680 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:51.426+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:51.680 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:51.427+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:52.651 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:52.379+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:52.651 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:52.379+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:52.651 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:52.379+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:52.651 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:52.379+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:52.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:52.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:52.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:52.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:52.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:52.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:52.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:52.429+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:52.652 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:52.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:52.652 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:52.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:52.652 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:52.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:52.652 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:52.374+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:53.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:04:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:04:52.650+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 11590 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:04:53.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:53.411+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:53.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:53.411+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:53.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:53.411+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:53.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:53.411+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:53.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:53.349+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:53.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:53.349+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:53.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:53.349+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:53.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:53.349+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:53.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:53.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:53.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:53.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:53.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:53.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:53.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:53.429+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:54.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:54.363+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:54.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:54.363+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:54.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:54.363+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:54.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:54.364+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:54.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:54.352+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:54.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:54.352+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:54.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:54.352+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:54.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:54.353+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:54.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:54.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:54.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:54.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:54.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:54.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:54.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:54.413+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:55.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:55.413+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:55.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:55.413+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:55.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:55.413+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:55.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:55.413+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:55.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:55.338+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:55.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:55.338+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:55.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:55.338+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:55.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:55.339+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:55.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:55.379+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:55.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:55.379+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:55.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:55.379+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:55.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:55.379+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:56.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:56.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:56.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:56.369+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:56.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:56.383+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:56.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:56.383+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:56.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:56.383+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:56.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:56.383+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:56.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:56.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:56.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:56.404+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:57.652 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:57.373+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:57.652 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:57.373+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:57.652 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:57.373+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:57.652 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:57.374+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:57.653 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:57.378+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:57.653 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:57.378+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:57.653 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:57.378+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:57.653 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:57.378+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:57.653 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:57.385+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:57.653 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:57.385+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:57.653 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:57.385+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:57.653 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:57.385+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:58.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:04:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:04:57.652+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 11684 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:04:58.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:58.347+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:58.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:58.347+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:58.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:58.347+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:58.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:58.347+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:58.377+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:58.377+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:58.377+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:58.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:58.378+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:58.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:58.423+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:58.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:58.423+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:58.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:58.423+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:58.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:58.423+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:59.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:59.321+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:59.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:59.321+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:59.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:59.321+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:04:59.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:04:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:04:59.322+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:59.377+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:59.377+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:59.377+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:04:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:04:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:04:59.377+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:04:59.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:59.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:59.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:59.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:59.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:59.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:04:59.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:04:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:04:59.405+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:00.416 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:00.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:00.416 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:00.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:00.416 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:00.337+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:00.416 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:00.338+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:00.417 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:00.373+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:00.417 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:00.373+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:00.417 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:00.373+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:00.417 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:00.374+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:00.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:00.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:00.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:00.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:00.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:00.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:00.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:00.416+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:01.680 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:01.380+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:01.680 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:01.380+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:01.680 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:01.380+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:01.680 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:01.380+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:01.680 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:01.349+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:01.680 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:01.349+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:01.680 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:01.349+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:01.680 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:01.349+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:01.681 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:01.375+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:01.681 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:01.375+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:01.681 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:01.375+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:01.681 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:01.376+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:02.654 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:02.354+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:02.655 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:02.354+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:02.655 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:02.354+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:02.655 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:02.354+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:02.655 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:02.419+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:02.655 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:02.419+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:02.655 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:02.419+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:02.655 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:02.419+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:02.656 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:02.346+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:02.656 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:02.346+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:02.656 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:02.346+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:02.656 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:02.346+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:03.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:05:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:05:02.654+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 11779 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:05:03.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:03.359+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:03.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:03.359+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:03.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:03.359+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:03.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:03.360+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:03.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:03.333+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:03.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:03.333+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:03.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:03.333+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:03.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:03.333+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:03.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:03.456+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:03.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:03.456+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:03.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:03.456+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:03.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:03.457+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:04.627 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:04.361+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:04.627 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:04.361+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:04.627 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:04.361+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:04.628 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:04.361+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:04.628 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:04.424+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:04.628 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:04.424+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:04.628 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:04.424+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:04.628 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:04.425+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:04.628 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:04.327+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:04.628 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:04.327+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:04.628 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:04.327+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:04.628 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:04.327+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:05.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:05.446+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:05.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:05.446+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:05.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:05.446+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:05.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:05.447+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:05.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:05.327+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:05.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:05.327+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:05.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:05.327+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:05.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:05.327+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:05.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:05.321+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:05.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:05.321+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:05.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:05.321+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:05.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:05.322+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:06.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:06.461+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:06.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:06.461+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:06.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:06.461+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:06.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:06.461+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:06.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:06.306+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:06.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:06.306+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:06.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:06.306+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:06.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:06.306+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:06.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:06.355+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:06.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:06.355+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:06.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:06.355+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:06.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:06.356+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:07.656 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:07.391+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:07.657 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:07.391+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:07.657 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:07.391+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:07.657 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:07.391+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:07.657 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:07.272+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:07.657 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:07.272+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:07.657 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:07.272+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:07.657 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:07.272+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:07.658 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:07.440+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:07.658 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:07.440+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:07.658 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:07.440+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:07.658 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:07.441+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:08.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:05:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:05:07.656+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 11872 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:05:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:08.348+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:08.348+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:08.348+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:08.348+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:08.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:08.294+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:08.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:08.294+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:08.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:08.294+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:08.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:08.294+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:08.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:08.457+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:08.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:08.457+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:08.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:08.457+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:08.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:08.457+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:09.431+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:09.431+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:09.431+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:09.431+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:09.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:09.327+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:09.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:09.327+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:09.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:09.327+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:09.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:09.328+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:09.351+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:09.351+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:09.351+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:09.351+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:10.397 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:10.312+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:10.397 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:10.312+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:10.397 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:10.312+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:10.397 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:10.312+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:10.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:10.455+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:10.455+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:10.455+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:10.455+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:10.397+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:10.397+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:10.397+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:10.397+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:11.680 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:11.288+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:11.680 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:11.288+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:11.680 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:11.288+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:11.681 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:11.288+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:11.681 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:11.502+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:11.681 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:11.502+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:11.681 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:11.502+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:11.681 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:11.502+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:11.681 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:11.442+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:11.681 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:11.442+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:11.681 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:11.442+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:11.682 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:11.443+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:12.658 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:12.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:12.658 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:12.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:12.658 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:12.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:12.658 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:12.490+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:12.659 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:12.442+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:12.659 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:12.442+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:12.659 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:12.442+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:12.659 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:12.442+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:12.659 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:12.268+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:12.659 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:12.268+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:12.659 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:12.268+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:12.660 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:12.268+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:13.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:05:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:05:12.658+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 11966 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:05:13.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:13.476+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:13.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:13.476+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:13.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:13.476+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:13.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:13.477+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:13.229+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:13.229+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:13.229+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:13.229+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:13.843 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:13.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:13.844 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:13.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:13.844 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:13.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:13.844 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:13.520+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:14.540 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:14.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:14.540 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:14.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:14.541 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:14.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:14.541 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:14.473+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:14.541 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:14.260+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:14.541 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:14.260+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:14.541 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:14.260+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:14.541 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:14.260+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:15.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:14.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:15.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:14.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:15.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:14.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:15.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:14.540+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:15.514+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:15.514+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:15.514+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:15.514+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:15.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:15.526+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:15.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:15.526+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:15.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:15.526+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:15.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:15.526+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:15.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:15.282+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:15.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:15.282+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:15.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:15.282+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:15.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:15.282+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:16.241+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:16.241+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:16.241+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:16.241+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:16.916 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:16.531+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:16.916 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:16.531+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:16.916 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:16.531+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:16.916 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:16.531+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:16.916 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:16.551+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:16.917 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:16.551+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:16.917 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:16.551+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:16.917 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:16.551+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:17.248+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:17.248+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:17.248+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:17.249+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:17.788 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:05:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:05:17.659+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 12060 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:05:17.788 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:17.517+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:17.788 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:17.517+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:17.788 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:17.517+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:17.788 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:17.518+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:17.789 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:17.532+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:17.789 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:17.532+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:17.789 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:17.532+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:17.789 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:17.533+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:18.530 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:18.256+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:18.530 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:18.256+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:18.530 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:18.256+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:18.530 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:18.256+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:18.537+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:18.537+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:18.537+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:18.537+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:19.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:18.530+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:19.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:18.530+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:19.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:18.530+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:19.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:18.530+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:19.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:19.536+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:19.536+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:19.536+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:19.537+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:19.488+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:19.488+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:19.488+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:19.488+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:19.295+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:19.295+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:19.295+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:19.295+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:20.298+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:20.298+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:20.298+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:20.298+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:21.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:20.581+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:21.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:20.581+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:21.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:20.581+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:21.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:20.581+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:20.534+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:20.534+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:20.534+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:20.534+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:21.578 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:21.273+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:21.578 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:21.273+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:21.578 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:21.273+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:21.578 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:21.273+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:22.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:21.596+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:21.596+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:21.596+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:21.596+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:21.578+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:21.578+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:21.578+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:21.578+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:22.599 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:22.254+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:22.599 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:22.254+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:22.599 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:22.254+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:22.599 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:22.254+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:23.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:05:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:05:22.661+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 12155 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:05:23.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:22.602+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:23.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:22.602+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:23.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:22.603+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:23.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:22.603+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:23.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:22.599+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:23.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:22.599+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:23.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:22.599+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:23.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:22.599+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:23.222+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:23.222+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:23.222+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:23.222+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:24.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:23.616+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:24.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:23.616+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:24.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:23.616+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:24.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:23.616+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:23.581+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:24.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:23.581+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:24.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:23.581+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:24.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:23.581+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:24.568 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:24.252+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:24.568 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:24.252+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:24.568 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:24.252+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:24.568 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:24.252+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:25.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:24.610+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:25.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:24.610+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:25.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:24.610+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:25.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:24.611+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:24.568+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:24.568+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:24.568+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:24.568+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:25.567 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:25.281+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:25.567 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:25.281+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:25.568 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:25.281+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:25.568 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:25.281+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:25.656+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:25.656+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:25.656+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:25.656+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:25.567+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:25.567+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:25.567+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:25.568+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:26.283+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:26.283+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:26.283+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:26.283+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:27.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:26.680+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:27.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:26.680+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:27.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:26.680+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:27.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:26.680+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:26.527+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:26.527+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:26.527+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:26.528+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:27.664 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:27.495+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:27.664 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:27.495+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:27.664 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:27.495+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:27.664 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:27.495+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:27.664 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:27.256+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:27.664 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:27.256+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:27.664 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:27.256+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:27.665 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:27.256+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:28.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:05:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:05:27.663+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 12248 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:05:28.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:27.675+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:28.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:27.675+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:28.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:27.675+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:28.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:27.676+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:28.217+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:28.217+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:28.217+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:28.217+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:29.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:28.687+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:29.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:28.687+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:29.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:28.687+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:29.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:28.687+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:28.522+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:28.522+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:28.522+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:28.522+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:29.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:29.249+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:29.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:29.249+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:29.490 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:29.249+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:29.490 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:29.249+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:29.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:29.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:29.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:29.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:29.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:29.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:29.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:29.680+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:29.489+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:29.489+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:29.489+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:29.489+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:30.245+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:30.245+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:30.245+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:30.245+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:30.662+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:30.662+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:30.662+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:30.662+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:31.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:30.519+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:31.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:30.519+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:31.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:30.519+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:31.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:30.519+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:31.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:31.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:31.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:31.198+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:31.689+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:31.689+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:31.689+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:31.690+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:31.522+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:31.522+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:31.522+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:31.522+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:32.229+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:32.229+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:32.229+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:32.229+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:32.790 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:05:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:05:32.665+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 12342 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:05:32.790 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:32.735+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:32.790 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:32.736+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:32.790 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:32.736+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:32.791 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:32.736+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:32.791 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:32.539+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:32.791 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:32.539+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:32.791 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:32.539+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:32.791 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:32.540+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:33.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:33.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:33.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:33.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:33.246+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:33.579+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:33.579+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:33.579+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:33.579+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:33.751+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:33.751+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:33.751+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:33.751+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:34.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:34.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:34.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:34.206+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:35.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:34.624+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:34.624+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:34.624+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:34.624+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:35.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:34.744+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:35.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:34.744+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:35.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:34.744+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:35.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:34.744+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:35.166+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:35.166+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:35.166+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:35.166+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:35.765+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:35.765+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:35.765+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:35.765+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:35.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:35.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:35.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:35.595+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:36.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:36.123+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:36.123+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:36.123+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:36.123+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:37.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:36.727+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:37.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:36.727+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:37.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:36.727+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:37.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:36.728+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:36.613+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:36.613+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:36.613+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:36.613+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:37.084+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:37.084+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:37.084+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:37.084+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:38.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:05:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:05:37.667+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 12437 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:05:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:37.583+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:37.583+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:37.583+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:37.583+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:37.762+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:37.762+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:37.762+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:37.762+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:38.420 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:38.093+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:38.420 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:38.093+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:38.420 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:38.093+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:38.420 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:38.093+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:39.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:38.794+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:38.794+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:38.794+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:38.795+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:38.555+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:38.555+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:38.555+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:38.555+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:39.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:39.082+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:39.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:39.082+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:39.082+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:39.082+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:39.844 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:39.518+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:39.844 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:39.518+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:39.844 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:39.518+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:39.844 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:39.518+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:40.144 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:40.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:40.144 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:40.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:40.144 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:40.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:40.144 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:40.088+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:40.145 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:39.844+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:40.145 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:39.844+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:40.145 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:39.844+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:40.145 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:39.844+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:40.802 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:40.507+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:40.802 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:40.507+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:40.802 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:40.507+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:40.802 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:40.508+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:41.112 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:40.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:41.113 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:40.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:41.113 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:40.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:41.113 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:40.801+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:41.491 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:41.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:41.491 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:41.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:41.491 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:41.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:41.491 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:41.112+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:41.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:41.519+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:41.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:41.519+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:41.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:41.519+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:41.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:41.519+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:42.154 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:41.769+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:42.154 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:41.769+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:42.154 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:41.769+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:42.154 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:41.769+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:42.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:42.153+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:42.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:42.153+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:42.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:42.153+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:42.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:42.153+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:42.793 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:05:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:05:42.668+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 12532 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:05:42.793 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:42.539+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:42.793 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:42.539+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:42.793 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:42.539+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:42.793 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:42.539+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:43.183 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:42.791+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:43.183 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:42.791+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:43.183 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:42.791+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:43.183 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:42.792+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:43.182+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:43.182+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:43.182+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:43.182+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:43.805 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:43.514+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:43.806 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:43.514+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:43.806 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:43.514+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:43.806 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:43.514+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:44.161 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:43.804+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:44.161 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:43.804+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:44.161 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:43.804+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:44.161 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:43.805+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:44.159+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:44.159+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:44.159+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:44.160+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:44.822 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:44.519+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:44.822 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:44.519+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:44.822 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:44.519+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:44.822 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:44.519+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:45.194 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:44.820+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:45.194 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:44.820+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:45.194 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:44.820+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:45.194 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:44.820+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:45.485 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:45.193+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:45.486 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:45.193+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:45.486 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:45.193+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:45.486 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:45.193+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:45.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:45.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:45.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:45.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:45.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:45.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:45.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:45.485+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:46.147 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:45.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:46.147 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:45.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:46.147 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:45.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:46.147 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:45.802+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:46.498 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:46.146+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:46.499 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:46.146+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:46.499 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:46.146+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:46.499 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:46.146+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:46.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:46.497+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:46.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:46.497+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:46.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:46.497+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:46.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:46.498+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:47.122 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:46.760+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:47.122 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:46.760+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:47.122 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:46.760+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:47.122 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:46.760+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:47.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:47.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:47.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:47.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:47.120+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:47.797 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:05:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:05:47.670+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 12626 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:05:47.797 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:47.519+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:47.797 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:47.519+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:47.798 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:47.519+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:47.798 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:47.520+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:47.798 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:47.720+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:47.798 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:47.720+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:47.798 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:47.720+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:47.798 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:47.720+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:48.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:48.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:48.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:48.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:48.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:48.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:48.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:48.088+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:49.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:48.709+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:49.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:48.709+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:49.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:48.709+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:49.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:48.709+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:48.545+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:48.545+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:48.545+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:48.546+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:49.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:49.099+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:49.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:49.099+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:49.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:49.099+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:49.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:49.099+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:50.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:49.507+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:49.507+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:49.507+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:49.508+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:50.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:49.688+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:50.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:49.688+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:50.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:49.688+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:50.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:49.688+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:50.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:50.062+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:50.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:50.062+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:50.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:50.062+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:50.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:50.062+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:50.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:50.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:50.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:50.679+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:50.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:50.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:50.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:50.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:50.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:50.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:50.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:50.464+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:51.424 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:51.018+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:51.425 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:51.018+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:51.425 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:51.018+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:51.425 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:51.018+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:51.683 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:51.651+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:51.683 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:51.651+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:51.683 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:51.651+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:51.684 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:51.651+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:51.684 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:51.424+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:51.684 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:51.424+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:51.684 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:51.424+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:51.684 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:51.424+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:52.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:52.000+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:52.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:52.000+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:52.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:52.000+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:52.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:52.000+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:52.673 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:52.410+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:52.674 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:52.410+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:52.674 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:52.410+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:52.674 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:52.410+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:52.974 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:05:52.672+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 12721 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:05:52.974 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:52.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:52.974 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:52.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:52.974 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:52.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:52.974 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:52.679+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:53.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:52.973+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:53.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:52.973+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:53.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:52.973+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:53.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:52.973+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:53.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:53.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:53.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:53.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:53.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:53.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:53.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:53.433+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:53.932+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:53.932+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:53.932+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:53.932+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:54.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:53.723+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:54.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:53.723+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:54.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:53.723+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:54.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:53.723+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:54.714 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:54.398+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:54.714 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:54.398+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:54.714 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:54.398+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:54.714 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:54.398+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:55.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:54.932+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:55.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:54.932+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:55.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:54.932+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:55.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:54.932+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:55.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:54.713+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:55.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:54.713+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:55.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:54.713+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:55.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:54.713+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:55.741 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:55.436+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:55.741 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:55.436+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:55.741 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:55.436+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:55.741 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:55.436+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:55.974+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:55.974+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:55.974+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:55.974+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:55.740+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:55.740+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:55.740+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:55.740+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:56.501 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:56.399+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:56.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:56.399+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:56.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:56.399+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:56.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:56.399+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:56.788 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:56.739+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:56.788 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:56.739+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:56.788 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:56.739+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:56.788 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:56.739+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:57.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:56.987+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:57.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:56.987+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:57.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:56.987+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:57.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:56.987+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:57.675 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:57.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:57.675 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:57.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:57.675 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:57.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:57.675 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:57.433+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:58.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:05:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:05:57.675+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 12816 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:05:58.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:57.768+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:58.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:57.768+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:58.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:57.768+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:58.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:57.768+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:58.453 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:58.001+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:58.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:58.001+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:58.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:58.001+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:58.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:58.002+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:58.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:58.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:58.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:58.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:58.454+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:59.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:58.983+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:59.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:58.983+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:59.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:58.983+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:05:59.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:05:58.983+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:59.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:58.803+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:59.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:58.803+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:59.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:58.803+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:05:59.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:58.803+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:05:59.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:59.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:59.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:59.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:59.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:05:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:05:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:05:59.429+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:00.145 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:00.027+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:00.145 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:00.027+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:00.145 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:00.027+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:00.145 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:00.027+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:00.145 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:59.831+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:00.145 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:59.831+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:00.145 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:59.831+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:00.146 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:05:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:05:59.831+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:00.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:00.449+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:00.449+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:00.449+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:00.450+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:01.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:00.870+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:01.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:00.870+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:01.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:00.870+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:01.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:00.871+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:01.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:01.069+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:01.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:01.069+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:01.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:01.069+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:01.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:01.069+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:01.684 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:01.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:01.684 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:01.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:01.684 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:01.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:01.684 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:01.405+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:01.900+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:01.900+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:01.900+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:01.901+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:02.405 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:02.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:02.405 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:02.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:02.405 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:02.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:02.406 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:02.112+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:02.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:02.405+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:02.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:02.405+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:02.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:02.405+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:02.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:02.406+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:02.995 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:06:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:06:02.677+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 12912 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:06:02.996 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:02.852+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:02.996 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:02.852+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:02.996 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:02.852+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:02.996 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:02.852+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:03.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:03.083+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:03.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:03.083+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:03.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:03.083+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:03.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:03.083+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:03.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:03.388+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:03.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:03.388+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:03.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:03.388+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:03.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:03.389+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:04.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:04.043+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:04.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:04.043+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:04.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:04.043+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:04.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:04.043+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:03.889+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:03.889+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:03.889+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:04.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:03.889+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:04.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:04.344+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:04.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:04.344+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:04.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:04.344+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:04.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:04.344+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:04.927+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:04.927+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:04.927+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:04.928+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:05.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:05.081+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:05.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:05.081+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:05.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:05.081+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:05.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:05.081+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:05.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:05.332+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:05.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:05.332+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:05.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:05.332+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:05.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:05.333+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:06.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:05.940+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:06.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:05.940+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:06.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:05.940+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:06.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:05.941+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:06.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:06.101+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:06.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:06.101+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:06.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:06.101+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:06.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:06.101+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:06.308+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:06.308+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:06.308+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:06.308+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:07.206 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:06.939+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:07.206 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:06.939+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:07.206 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:06.939+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:07.206 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:06.940+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:07.206 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:07.079+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:07.207 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:07.079+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:07.207 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:07.079+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:07.207 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:07.079+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:07.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:07.277+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:07.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:07.277+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:07.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:07.277+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:07.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:07.278+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:08.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:06:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:06:07.679+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 13006 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:06:08.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:07.926+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:08.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:07.926+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:08.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:07.926+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:08.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:07.926+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:08.268 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:08.123+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:08.268 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:08.123+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:08.268 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:08.123+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:08.268 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:08.123+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:08.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:08.268+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:08.268+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:08.268+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:08.268+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:09.229 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:08.941+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:09.229 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:08.941+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:09.229 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:08.941+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:09.229 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:08.942+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:09.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:09.119+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:09.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:09.119+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:09.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:09.119+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:09.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:09.119+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:09.501 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:09.228+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:09.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:09.228+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:09.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:09.228+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:09.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:09.229+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:10.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:10.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:10.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:10.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:10.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:10.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:10.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:10.115+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:10.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:09.943+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:10.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:09.943+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:10.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:09.943+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:10.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:09.944+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:10.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:10.267+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:10.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:10.267+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:10.267+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:10.268+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:11.235 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:11.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:11.236 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:11.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:11.236 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:11.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:11.236 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:11.113+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:11.236 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:10.925+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:11.236 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:10.925+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:11.236 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:10.925+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:11.236 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:10.926+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:11.501 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:11.235+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:11.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:11.235+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:11.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:11.235+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:11.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:11.236+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:12.187 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:11.936+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:12.187 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:11.936+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:12.187 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:11.936+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:12.187 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:11.936+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:12.188 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:12.093+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:12.188 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:12.093+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:12.188 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:12.093+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:12.188 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:12.093+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:12.501 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:12.187+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:12.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:12.187+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:12.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:12.187+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:12.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:12.187+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:12.933 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:06:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:06:12.681+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 13101 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:06:13.208 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:12.933+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:13.209 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:12.933+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:13.209 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:12.933+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:13.209 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:12.933+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:13.209 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:13.092+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:13.210 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:13.092+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:13.210 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:13.092+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:13.210 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:13.092+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:13.501 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:13.208+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:13.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:13.208+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:13.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:13.208+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:13.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:13.208+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:14.210 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:14.064+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:14.210 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:14.064+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:14.210 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:14.064+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:14.210 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:14.064+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:14.210 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:13.887+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:14.210 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:13.887+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:14.210 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:13.887+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:14.210 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:13.887+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:14.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:14.210+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:14.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:14.210+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:14.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:14.210+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:14.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:14.210+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:15.235 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:15.094+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:15.235 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:15.094+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:15.235 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:15.094+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:15.235 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:15.094+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:15.235 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:14.933+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:15.236 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:14.933+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:15.236 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:14.933+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:15.236 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:14.933+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:15.501 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:15.235+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:15.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:15.235+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:15.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:15.235+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:15.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:15.235+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:16.236 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:16.096+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:16.236 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:16.096+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:16.236 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:16.096+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:16.236 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:16.096+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:16.236 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:15.962+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:16.236 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:15.962+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:16.236 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:15.962+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:16.237 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:15.963+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:16.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:16.235+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:16.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:16.235+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:16.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:16.235+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:16.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:16.236+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:17.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:16.998+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:17.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:16.998+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:17.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:16.998+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:17.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:16.999+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:17.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:17.210+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:17.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:17.210+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:17.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:17.210+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:17.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:17.211+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:17.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:17.058+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:17.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:17.058+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:17.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:17.058+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:17.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:17.058+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:18.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:06:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:06:17.683+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 13195 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:06:18.356 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:18.029+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:18.356 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:18.029+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:18.356 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:18.029+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:18.356 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:18.030+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:18.356 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:18.256+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:18.356 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:18.256+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:18.356 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:18.256+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:18.357 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:18.257+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:18.357 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:18.082+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:18.357 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:18.082+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:18.357 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:18.082+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:18.357 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:18.082+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:19.304 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:19.053+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:19.304 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:19.053+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:19.304 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:19.053+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:19.305 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:19.053+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:19.305 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:19.011+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:19.305 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:19.011+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:19.305 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:19.011+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:19.306 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:19.011+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:19.303+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:19.303+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:19.303+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:19.304+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:20.319 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:20.084+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:20.319 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:20.084+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:20.319 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:20.084+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:20.319 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:20.084+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:20.320 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:20.004+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:20.320 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:20.004+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:20.320 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:20.004+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:20.320 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:20.005+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:20.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:20.319+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:20.319+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:20.319+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:20.319+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:21.359 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:21.021+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:21.359 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:21.021+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:21.359 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:21.021+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:21.359 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:21.021+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:21.359 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:21.059+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:21.359 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:21.059+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:21.359 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:21.059+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:21.360 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:21.059+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:21.685 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:21.358+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:21.685 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:21.358+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:21.685 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:21.358+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:21.686 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:21.359+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:22.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:22.001+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:22.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:22.001+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:22.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:22.001+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:22.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:22.001+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:22.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:22.075+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:22.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:22.075+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:22.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:22.075+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:22.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:22.076+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:22.685 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:22.338+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:22.685 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:22.338+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:22.685 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:22.338+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:22.686 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:22.338+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:22.972 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:06:22.685+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 13291 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:06:23.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:22.972+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:23.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:22.972+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:23.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:22.972+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:23.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:22.972+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:23.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:23.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:23.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:23.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:23.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:23.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:23.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:23.089+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:23.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:23.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:23.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:23.339+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:24.387 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:24.011+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:24.387 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:24.011+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:24.388 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:24.011+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:24.388 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:24.011+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:24.388 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:24.102+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:24.388 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:24.102+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:24.388 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:24.102+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:24.389 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:24.102+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:24.387+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:24.387+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:24.387+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:24.387+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:25.404 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:25.046+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:25.404 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:25.046+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:25.405 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:25.046+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:25.405 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:25.047+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:25.405 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:25.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:25.405 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:25.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:25.405 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:25.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:25.405 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:25.120+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:25.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:25.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:25.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:25.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:25.404+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:26.393 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:26.046+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:26.393 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:26.046+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:26.393 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:26.046+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:26.393 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:26.047+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:26.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:26.073+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:26.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:26.073+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:26.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:26.073+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:26.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:26.073+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:26.393+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:26.393+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:26.393+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:26.393+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:27.389 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:27.080+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:27.389 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:27.080+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:27.389 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:27.080+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:27.389 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:27.080+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:27.390 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:27.008+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:27.390 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:27.008+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:27.390 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:27.008+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:27.390 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:27.008+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:27.687 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:27.389+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:27.688 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:27.389+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:27.688 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:27.389+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:27.688 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:27.389+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:28.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:06:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:06:27.687+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 13384 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:06:28.353 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:28.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:28.354 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:28.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:28.354 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:28.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:28.354 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:28.116+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:28.354 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:28.006+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:28.354 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:28.006+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:28.354 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:28.006+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:28.354 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:28.007+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:28.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:28.353+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:28.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:28.353+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:28.353+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:28.354+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:29.359 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:29.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:29.360 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:29.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:29.360 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:29.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:29.360 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:29.115+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:29.360 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:29.009+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:29.360 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:29.009+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:29.360 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:29.009+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:29.360 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:29.009+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:29.403 INFO:teuthology.orchestra.run.smithi052.stderr:2023-12-16T21:06:29.401+0000 7fd7c9722700 0 monclient(hunting): authenticate timed out after 300 2023-12-16T21:06:29.403 INFO:teuthology.orchestra.run.smithi052.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-16T21:06:29.670 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:29.359+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:29.670 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:29.359+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:29.670 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:29.359+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:29.670 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:29.360+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:29.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:06:29.703 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1351, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2023-12-16T21:06:29.785 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e4bfae22d6f94512a47a4bb12e1b73b7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1351, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2023-12-16T21:06:29.789 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-12-16T21:06:29.803 INFO:tasks.kclient:Unmounting kernel clients... 2023-12-16T21:06:29.804 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T21:06:29.804 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:06:29.832 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T21:06:29.832 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:06:29.894 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:06:30.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:30.057+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:30.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:30.058+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:30.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:30.058+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:30.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:30.058+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:30.154+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:30.154+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:30.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:30.154+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:30.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:30.154+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:30.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:30.405+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:30.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:30.405+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:30.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:30.405+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:30.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:30.405+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:31.395 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:31.058+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:31.395 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:31.058+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:31.396 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:31.058+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:31.396 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:31.058+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:31.396 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:31.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:31.396 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:31.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:31.396 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:31.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:31.396 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:31.113+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:31.686 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:31.395+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:31.687 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:31.395+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:31.687 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:31.395+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:31.687 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:31.395+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:32.408 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:32.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:32.408 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:32.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:32.408 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:32.082+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:32.409 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:32.082+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:32.409 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:32.104+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:32.409 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:32.104+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:32.409 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:32.104+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:32.409 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:32.104+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:32.690 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:32.408+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:32.690 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:32.408+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:32.690 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:32.408+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:32.691 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:32.408+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:33.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:06:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:06:32.689+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 13480 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:06:33.389 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:33.101+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:33.389 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:33.101+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:33.389 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:33.101+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:33.390 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:33.101+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:33.390 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:33.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:33.390 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:33.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:33.391 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:33.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:33.391 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:33.067+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:33.715 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:33.388+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:33.715 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:33.388+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:33.716 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:33.388+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:33.716 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:33.388+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:34.428 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:34.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:34.429 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:34.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:34.429 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:34.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:34.429 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:34.074+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:34.429 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:34.147+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:34.429 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:34.147+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:34.430 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:34.147+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:34.430 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:34.148+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:34.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:34.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:34.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:34.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:34.428+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:35.414 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:35.150+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:35.414 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:35.150+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:35.414 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:35.150+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:35.414 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:35.151+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:35.415 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:35.080+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:35.415 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:35.080+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:35.415 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:35.080+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:35.415 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:35.081+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:35.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:35.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:35.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:35.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:35.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:35.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:35.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:35.414+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:36.397 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:36.153+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:36.397 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:36.153+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:36.397 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:36.153+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:36.397 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:36.153+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:36.398 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:36.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:36.398 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:36.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:36.398 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:36.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:36.399 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:36.089+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:36.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:36.396+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:36.396+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:36.396+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:36.397+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:37.430 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:37.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:37.430 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:37.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:37.430 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:37.139+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:37.430 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:37.140+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:37.431 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:37.065+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:37.431 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:37.065+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:37.431 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:37.065+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:37.431 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:37.065+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:37.693 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:37.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:37.693 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:37.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:37.693 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:37.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:37.693 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:37.430+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:38.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:06:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:06:37.691+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 13574 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:06:38.462 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:38.169+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:38.463 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:38.169+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:38.463 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:38.169+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:38.463 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:38.169+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:38.463 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:38.055+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:38.463 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:38.055+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:38.463 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:38.055+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:38.464 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:38.056+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:38.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:38.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:38.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:38.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:38.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:38.462+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:39.465 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:39.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:39.465 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:39.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:39.465 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:39.178+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:39.465 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:39.178+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:39.466 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:39.050+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:39.466 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:39.050+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:39.466 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:39.050+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:39.466 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:39.050+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:39.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:39.465+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:39.465+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:39.465+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:39.465+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:40.149 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:40.009+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:40.149 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:40.009+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:40.149 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:40.009+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:40.149 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:40.010+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:40.453 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:40.221+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:40.453 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:40.221+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:40.453 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:40.221+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:40.454 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:40.222+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:40.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:40.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:40.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:40.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:40.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:40.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:40.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:40.453+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:41.443 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:41.258+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:41.443 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:41.258+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:41.443 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:41.258+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:41.443 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:41.258+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:41.443 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:41.059+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:41.444 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:41.059+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:41.444 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:41.059+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:41.444 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:41.059+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:41.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:41.442+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:41.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:41.442+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:41.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:41.442+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:41.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:41.443+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:42.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:42.233+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:42.417 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:42.233+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:42.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:42.233+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:42.418 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:42.233+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:42.418 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:42.046+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:42.418 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:42.046+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:42.418 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:42.046+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:42.419 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:42.046+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:42.694 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:42.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:42.694 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:42.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:42.694 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:42.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:42.694 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:42.417+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:43.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:06:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:06:42.693+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 13671 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:06:43.425 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:43.224+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:43.425 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:43.224+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:43.425 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:43.224+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:43.426 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:43.224+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:43.426 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:43.076+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:43.426 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:43.076+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:43.426 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:43.076+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:43.426 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:43.076+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:43.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:43.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:43.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:43.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:43.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:43.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:43.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:43.425+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:44.414 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:44.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:44.414 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:44.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:44.414 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:44.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:44.414 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:44.071+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:44.415 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:44.238+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:44.415 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:44.238+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:44.415 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:44.238+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:44.415 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:44.239+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:44.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:44.414+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:44.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:44.414+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:44.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:44.414+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:44.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:44.414+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:45.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:45.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:45.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:45.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:45.456 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:45.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:45.456 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:45.074+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:45.456 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:45.287+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:45.456 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:45.287+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:45.456 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:45.287+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:45.457 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:45.287+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:45.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:45.455+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:45.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:45.455+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:45.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:45.455+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:45.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:45.455+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:46.257 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:46.052+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:46.257 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:46.052+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:46.258 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:46.052+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:46.258 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:46.052+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:46.257+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:46.257+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:46.257+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:46.258+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:46.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:46.486+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:46.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:46.486+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:46.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:46.486+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:46.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:46.486+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:47.486 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:47.297+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:47.487 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:47.297+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:47.487 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:47.297+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:47.487 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:47.298+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:47.487 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:47.081+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:47.487 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:47.081+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:47.487 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:47.081+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:47.488 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:47.081+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:47.751 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:06:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:06:47.695+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 13763 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:06:47.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:47.487+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:47.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:47.487+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:47.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:47.487+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:47.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:47.487+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:48.332 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:48.080+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:48.332 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:48.080+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:48.332 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:48.080+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:48.332 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:48.080+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:48.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:48.331+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:48.331+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:48.331+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:48.332+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:48.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:48.499+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:48.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:48.499+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:48.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:48.499+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:48.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:48.499+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:49.333 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:49.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:49.334 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:49.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:49.334 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:49.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:49.334 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:49.049+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:49.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:49.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:49.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:49.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:49.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:49.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:49.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:49.485+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:49.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:49.333+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:49.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:49.333+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:49.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:49.333+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:49.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:49.334+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:50.287 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:50.035+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:50.287 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:50.035+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:50.287 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:50.036+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:50.287 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:50.036+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:50.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:50.498+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:50.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:50.498+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:50.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:50.498+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:50.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:50.498+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:50.287+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:50.287+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:50.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:50.287+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:50.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:50.287+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:51.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:51.001+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:51.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:51.001+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:51.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:51.001+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:51.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:51.001+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:51.688 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:51.472+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:51.689 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:51.472+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:51.689 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:51.472+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:51.689 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:51.473+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:51.689 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:51.318+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:51.689 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:51.318+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:51.689 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:51.318+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:51.690 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:51.318+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:52.366 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:52.009+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:52.367 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:52.009+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:52.367 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:52.009+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:52.367 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:52.009+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:52.698 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:52.511+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:52.698 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:52.511+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:52.698 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:52.511+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:52.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:52.512+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:52.699 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:52.366+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:52.699 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:52.366+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:52.699 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:52.366+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:52.699 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:52.367+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:53.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:06:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:06:52.698+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 13858 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:06:53.367 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:53.018+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:53.368 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:53.018+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:53.368 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:53.018+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:53.368 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:53.018+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:53.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:53.367+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:53.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:53.367+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:53.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:53.367+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:53.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:53.368+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:53.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:53.522+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:53.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:53.522+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:53.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:53.522+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:53.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:53.522+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:54.393 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:54.039+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:54.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:54.039+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:54.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:54.039+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:54.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:54.039+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:54.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:54.393+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:54.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:54.393+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:54.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:54.393+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:54.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:54.394+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:54.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:54.564+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:54.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:54.564+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:54.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:54.564+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:54.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:54.564+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:55.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:54.993+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:55.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:54.993+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:55.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:54.993+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:55.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:54.993+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:55.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:55.373+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:55.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:55.373+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:55.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:55.373+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:55.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:55.373+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:55.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:55.608+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:55.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:55.608+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:55.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:55.608+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:55.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:55.608+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:56.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:56.043+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:56.395 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:56.043+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:56.395 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:56.043+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:56.395 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:56.043+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:56.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:56.593+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:56.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:56.593+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:56.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:56.593+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:56.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:56.593+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:56.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:56.395+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:56.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:56.395+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:56.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:56.395+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:56.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:56.395+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:57.432 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:57.086+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:57.432 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:57.086+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:57.432 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:57.086+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:57.432 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:57.086+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:57.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:57.617+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:57.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:57.617+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:57.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:57.617+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:57.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:57.617+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:57.700 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:57.432+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:57.700 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:57.432+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:57.700 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:57.432+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:57.700 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:57.432+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:58.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:06:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:06:57.700+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 13951 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:06:58.458 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:58.042+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:58.458 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:58.042+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:58.458 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:58.042+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:58.459 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:58.042+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:58.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:58.458+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:58.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:58.458+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:58.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:58.458+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:58.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:58.458+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:58.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:58.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:58.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:58.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:58.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:58.630+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:59.462 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:59.061+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:59.462 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:59.061+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:59.463 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:59.061+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:06:59.463 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:06:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:06:59.061+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:59.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:59.462+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:59.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:59.462+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:59.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:59.462+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:06:59.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:06:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:06:59.462+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:06:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:59.644+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:59.644+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:59.644+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:06:59.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:06:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:06:59.644+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:00.472 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:00.061+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:00.473 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:00.061+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:00.473 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:00.061+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:00.473 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:00.062+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:00.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:00.470+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:00.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:00.470+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:00.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:00.470+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:00.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:00.470+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:00.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:00.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:00.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:00.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:00.610+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:01.363 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:01.087+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:01.364 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:01.087+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:01.364 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:01.087+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:01.364 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:01.087+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:01.689 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:01.478+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:01.689 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:01.478+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:01.690 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:01.478+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:01.690 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:01.478+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:01.690 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:01.592+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:01.690 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:01.592+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:01.690 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:01.592+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:01.690 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:01.592+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:02.472 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:02.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:02.472 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:02.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:02.472 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:02.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:02.472 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:02.129+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:02.723 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:07:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:07:02.701+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 14047 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:07:02.723 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:02.471+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:02.723 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:02.471+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:02.723 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:02.471+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:02.723 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:02.471+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:02.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:02.576+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:02.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:02.576+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:02.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:02.576+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:02.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:02.576+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:03.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:03.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:03.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:03.179+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:04.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:03.502+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:03.502+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:03.502+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:03.502+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:03.560+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:03.560+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:04.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:03.560+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:04.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:03.560+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:04.491 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:04.223+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:04.491 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:04.223+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:04.491 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:04.223+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:04.491 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:04.223+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:04.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:04.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:04.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:04.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:04.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:04.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:04.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:04.490+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:04.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:04.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:04.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:04.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:04.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:04.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:04.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:04.512+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:05.470 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:05.177+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:05.470 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:05.177+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:05.470 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:05.177+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:05.470 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:05.177+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:05.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:05.468+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:05.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:05.468+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:05.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:05.469+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:05.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:05.469+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:05.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:05.524+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:05.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:05.524+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:05.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:05.524+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:05.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:05.524+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:06.433 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:06.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:06.433 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:06.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:06.434 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:06.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:06.434 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:06.134+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:06.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:06.433+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:06.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:06.433+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:06.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:06.433+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:06.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:06.433+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:06.506+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:06.506+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:06.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:06.506+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:06.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:06.506+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:07.440 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:07.090+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:07.441 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:07.090+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:07.441 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:07.090+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:07.441 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:07.090+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:07.704 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:07.493+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:07.705 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:07.493+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:07.705 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:07.493+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:07.705 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:07.493+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:07.705 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:07.439+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:07.705 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:07.439+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:07.706 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:07.439+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:07.706 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:07.439+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:08.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:07:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:07:07.703+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 14139 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:07:08.420 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:08.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:08.420 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:08.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:08.420 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:08.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:08.421 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:08.133+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:08.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:08.515+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:08.515+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:08.515+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:08.515+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:08.420+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:08.420+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:08.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:08.420+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:08.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:08.420+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:09.471 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:09.158+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:09.471 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:09.158+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:09.471 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:09.158+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:09.471 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:09.158+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:09.522+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:09.522+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:09.522+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:09.522+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:09.470+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:09.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:09.470+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:09.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:09.470+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:09.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:09.470+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:10.488 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:10.200+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:10.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:10.200+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:10.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:10.200+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:10.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:10.201+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:10.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:10.501+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:10.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:10.501+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:10.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:10.501+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:10.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:10.501+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:10.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:10.486+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:10.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:10.486+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:10.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:10.486+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:10.756 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:10.487+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:11.161+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:11.161+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:11.161+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:11.161+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:11.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:11.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:11.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:11.512+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:12.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:11.527+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:11.527+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:11.527+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:11.527+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:12.486 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:12.132+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:12.487 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:12.132+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:12.487 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:12.132+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:12.487 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:12.132+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:12.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:07:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:07:12.705+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 14234 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:07:12.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:12.486+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:12.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:12.486+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:12.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:12.486+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:12.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:12.486+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:12.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:12.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:12.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:12.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:12.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:12.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:12.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:12.558+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:13.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:13.144+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:13.144+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:13.144+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:13.144+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:14.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:13.542+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:14.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:13.542+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:14.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:13.542+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:14.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:13.542+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:14.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:13.515+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:14.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:13.515+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:14.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:13.515+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:14.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:13.516+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:14.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:14.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:14.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:14.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:14.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:14.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:14.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:14.133+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:14.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:14.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:14.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:14.585+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:15.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:14.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:15.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:14.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:15.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:14.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:15.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:14.563+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:15.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:15.119+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:15.119+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:15.119+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:15.119+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:16.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:15.561+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:15.561+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:15.561+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:15.561+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:16.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:15.566+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:16.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:15.566+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:15.566+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:15.566+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:16.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:16.152+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:16.152+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:16.152+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:16.152+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:17.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:16.542+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:16.542+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:16.542+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:16.542+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:16.614+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:16.614+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:17.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:16.614+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:17.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:16.614+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:17.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:17.147+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:17.147+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:17.147+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:17.147+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:17.834 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:07:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:07:17.708+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 14328 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:07:17.834 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:17.610+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:17.834 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:17.610+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:17.835 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:17.610+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:17.835 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:17.610+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:17.835 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:17.553+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:17.835 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:17.553+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:17.835 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:17.553+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:17.835 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:17.554+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:18.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:18.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:18.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:18.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:18.179+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:19.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:18.651+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:19.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:18.651+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:19.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:18.651+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:19.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:18.652+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:19.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:18.534+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:19.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:18.534+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:19.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:18.534+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:19.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:18.534+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:19.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:19.218+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:19.218+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:19.218+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:19.218+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:19.796 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:19.602+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:19.796 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:19.602+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:19.796 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:19.602+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:19.797 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:19.602+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:19.797 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:19.516+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:19.797 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:19.516+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:19.797 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:19.516+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:19.797 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:19.516+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:20.475 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:20.238+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:20.475 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:20.238+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:20.476 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:20.238+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:20.476 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:20.239+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:20.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:20.627+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:20.627+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:20.627+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:20.628+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:20.475+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:20.475+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:20.475+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:20.475+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:21.483 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:21.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:21.483 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:21.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:21.483 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:21.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:21.483 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:21.206+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:21.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:21.483+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:21.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:21.483+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:21.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:21.483+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:21.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:21.483+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:21.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:21.617+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:21.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:21.617+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:21.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:21.617+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:21.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:21.618+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:22.499 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:22.217+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:22.499 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:22.217+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:22.499 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:22.217+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:22.499 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:22.217+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:22.751 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:07:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:07:22.711+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 14425 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:07:22.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:22.657+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:22.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:22.657+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:22.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:22.657+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:22.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:22.658+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:22.499+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:22.499+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:22.499+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:22.500+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:23.530 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:23.259+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:23.530 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:23.259+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:23.530 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:23.259+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:23.530 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:23.259+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:24.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:23.697+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:24.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:23.697+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:24.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:23.697+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:24.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:23.698+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:24.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:23.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:24.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:23.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:24.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:23.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:24.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:23.530+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:24.250+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:24.250+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:24.250+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:24.251+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:25.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:24.707+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:25.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:24.707+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:25.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:24.707+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:25.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:24.708+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:24.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:24.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:25.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:24.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:25.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:24.527+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:25.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:25.227+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:25.227+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:25.227+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:25.227+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:26.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:25.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:25.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:25.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:25.679+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:25.515+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:25.515+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:26.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:25.515+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:26.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:25.515+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:26.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:26.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:26.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:26.189+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:27.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:26.703+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:27.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:26.703+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:27.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:26.703+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:27.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:26.703+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:26.562+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:27.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:26.562+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:27.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:26.562+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:27.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:26.563+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:27.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:27.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:27.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:27.189+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:28.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:07:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:07:27.713+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 14518 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:07:28.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:27.675+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:28.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:27.675+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:28.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:27.675+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:28.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:27.676+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:28.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:27.588+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:28.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:27.588+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:28.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:27.588+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:28.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:27.589+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:28.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:28.168+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:28.168+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:28.168+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:28.168+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:29.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:28.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:29.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:28.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:29.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:28.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:29.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:28.638+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:28.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:29.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:28.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:29.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:28.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:29.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:28.614+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:29.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:29.140+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:29.140+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:29.140+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:29.141+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:30.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:29.634+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:30.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:29.634+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:30.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:29.634+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:30.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:29.635+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:29.652+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:29.652+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:30.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:29.652+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:30.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:29.652+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:30.148+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:30.148+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:30.148+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:30.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:30.148+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:31.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:30.597+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:30.597+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:30.597+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:30.597+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:30.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:30.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:30.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:31.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:30.611+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:31.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:31.111+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:31.111+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:31.111+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:31.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:31.111+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:31.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:31.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:31.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:31.576+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:31.575+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:32.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:31.575+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:32.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:31.575+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:32.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:31.575+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:32.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:32.108+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:32.108+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:32.108+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:32.108+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:32.848 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:07:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:07:32.715+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 14615 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:07:32.848 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:32.568+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:32.848 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:32.568+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:32.848 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:32.568+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:32.849 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:32.569+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:32.849 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:32.600+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:32.849 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:32.600+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:32.851 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:32.600+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:32.851 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:32.600+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:33.135+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:33.135+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:33.135+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:33.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:33.136+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:33.607+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:33.607+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:33.607+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:33.607+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:33.632+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:34.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:33.632+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:34.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:33.632+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:34.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:33.632+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:34.181+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:34.181+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:34.181+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:34.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:34.181+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:34.663+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:34.663+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:34.663+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:34.663+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:35.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:34.650+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:35.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:34.650+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:35.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:34.650+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:35.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:34.651+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:35.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:35.214+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:35.214+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:35.214+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:35.214+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:35.655+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:35.655+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:35.655+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:35.655+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:35.662+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:36.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:35.662+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:36.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:35.662+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:36.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:35.662+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:36.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:36.227+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:36.227+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:36.227+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:36.227+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:37.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:36.682+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:37.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:36.682+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:37.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:36.682+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:37.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:36.683+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:36.697+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:37.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:36.697+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:37.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:36.697+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:37.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:36.698+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:37.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:37.219+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:37.219+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:37.219+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:37.219+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:38.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:07:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:07:37.717+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 14706 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:07:38.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:37.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:38.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:37.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:38.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:37.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:38.006 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:37.707+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:38.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:37.641+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:38.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:37.641+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:38.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:37.641+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:38.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:37.641+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:38.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:38.235+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:38.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:38.235+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:38.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:38.235+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:38.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:38.235+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:39.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:38.720+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:38.720+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:38.720+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:38.721+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:38.678+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:38.678+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:38.678+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:39.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:38.678+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:39.241+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:39.241+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:39.241+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:39.242+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:40.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:39.724+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:40.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:39.724+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:40.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:39.724+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:40.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:39.724+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:39.705+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:39.705+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:39.705+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:39.706+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:40.214+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:40.214+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:40.214+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:40.214+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:41.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:40.712+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:41.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:40.712+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:41.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:40.712+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:41.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:40.713+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:40.675+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:40.675+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:40.675+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:40.676+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:41.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:41.232+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:41.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:41.232+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:41.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:41.232+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:41.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:41.232+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:42.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:41.640+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:41.640+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:41.640+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:41.640+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:41.701+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:41.701+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:41.701+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:42.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:41.701+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:42.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:42.247+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:42.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:42.247+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:42.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:42.247+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:42.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:42.247+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:43.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:07:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:07:42.719+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 14804 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:07:43.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:42.663+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:43.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:42.663+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:43.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:42.663+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:43.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:42.664+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:43.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:42.634+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:43.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:42.634+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:43.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:42.634+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:43.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:42.635+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:43.217+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:43.217+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:43.217+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:43.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:43.217+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:44.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:43.628+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:44.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:43.628+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:44.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:43.628+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:44.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:43.629+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:43.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:43.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:43.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:44.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:43.611+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:44.184+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:44.184+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:44.184+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:44.184+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:44.884 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:44.582+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:44.884 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:44.582+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:44.884 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:44.582+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:44.884 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:44.583+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:44.885 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:44.582+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:44.885 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:44.582+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:44.885 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:44.582+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:44.885 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:44.583+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:45.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:45.160+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:45.160+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:45.160+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:45.160+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:46.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:45.623+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:46.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:45.623+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:46.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:45.623+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:46.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:45.624+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:45.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:45.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:46.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:45.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:46.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:45.585+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:46.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:46.176+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:46.176+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:46.176+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:46.176+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:47.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:46.628+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:47.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:46.628+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:47.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:46.628+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:47.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:46.629+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:46.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:47.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:46.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:47.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:46.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:47.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:46.559+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:47.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:47.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:47.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:47.215+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:47.847 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:07:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:07:47.722+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 14898 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:07:47.847 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:47.629+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:47.847 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:47.629+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:47.847 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:47.629+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:47.847 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:47.629+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:47.848 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:47.566+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:47.848 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:47.566+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:47.848 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:47.566+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:47.848 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:47.567+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:48.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:48.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:48.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:48.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:48.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:48.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:48.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:48.215+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:49.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:48.675+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:49.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:48.675+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:49.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:48.675+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:49.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:48.676+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:48.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:49.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:48.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:49.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:48.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:49.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:48.571+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:49.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:49.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:49.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:49.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:49.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:49.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:49.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:49.179+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:50.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:49.703+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:50.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:49.703+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:50.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:49.703+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:50.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:49.704+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:49.554+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:49.554+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:50.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:49.554+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:50.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:49.555+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:50.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:50.201+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:50.201+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:50.201+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:50.202+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:51.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:50.712+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:51.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:50.712+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:51.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:50.712+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:51.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:50.712+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:50.596+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:50.596+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:51.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:50.596+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:51.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:50.596+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:51.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:51.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:51.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:51.192+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:51.714+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:51.714+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:51.714+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:51.715+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:52.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:51.555+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:52.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:51.555+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:52.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:51.555+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:52.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:51.556+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:52.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:52.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:52.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:52.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:52.179+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:52.851 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:07:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:07:52.724+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 14995 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:07:52.851 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:52.745+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:52.851 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:52.745+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:52.851 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:52.745+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:52.852 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:52.746+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:52.852 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:52.510+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:52.852 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:52.510+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:52.852 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:52.510+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:52.852 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:52.510+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:53.219+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:53.219+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:53.219+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:53.219+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:53.521+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:53.521+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:53.521+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:53.521+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:54.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:53.716+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:54.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:53.716+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:54.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:53.716+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:54.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:53.716+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:54.548 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:54.262+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:54.548 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:54.262+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:54.548 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:54.262+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:54.548 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:54.262+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:55.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:54.671+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:55.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:54.671+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:55.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:54.671+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:55.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:54.672+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:54.547+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:54.547+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:55.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:54.547+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:55.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:54.548+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:55.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:55.245+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:55.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:55.245+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:55.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:55.245+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:55.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:55.245+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:56.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:55.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:55.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:55.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:55.512+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:56.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:55.684+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:56.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:55.684+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:56.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:55.684+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:56.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:55.684+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:56.557 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:56.280+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:56.558 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:56.280+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:56.558 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:56.280+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:56.558 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:56.280+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:56.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:56.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:56.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:57.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:56.558+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:56.642+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:56.642+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:56.642+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:56.642+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:57.517 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:57.264+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:57.517 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:57.264+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:57.517 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:57.264+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:57.517 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:57.264+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:57.850 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:07:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:07:57.726+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 15085 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:07:57.851 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:57.515+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:57.851 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:57.515+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:57.851 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:57.515+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:57.851 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:57.515+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:57.851 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:57.684+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:57.852 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:57.684+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:57.852 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:57.685+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:57.852 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:57.685+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:58.683 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:58.470+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:58.683 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:58.470+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:58.683 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:58.470+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:58.683 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:58.470+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:58.683 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:58.259+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:58.684 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:58.259+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:58.684 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:58.259+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:58.684 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:58.259+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:58.683+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:58.683+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:58.683+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:07:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:58.683+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:59.501 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:59.466+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:59.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:59.466+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:59.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:59.466+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:07:59.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:07:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:07:59.466+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:07:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:59.248+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:59.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:59.248+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:59.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:59.248+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:07:59.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:07:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:07:59.248+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:00.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:59.705+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:00.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:59.705+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:00.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:59.705+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:00.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:07:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:07:59.705+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:00.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:00.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:00.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:00.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:00.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:00.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:00.490 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:00.206+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:00.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:00.690+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:00.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:00.690+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:00.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:00.690+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:00.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:00.690+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:00.489+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:00.489+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:00.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:00.489+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:00.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:00.489+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:01.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:01.205+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:01.205+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:01.205+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:01.205+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:02.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:01.649+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:01.649+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:01.649+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:01.649+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:01.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:01.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:01.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:02.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:01.529+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:02.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:02.186+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:02.186+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:02.186+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:02.186+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:02.853 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:08:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:08:02.729+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 15169 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:08:02.854 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:02.617+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:02.854 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:02.617+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:02.854 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:02.617+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:02.854 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:02.617+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:02.854 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:02.576+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:02.855 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:02.576+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:02.855 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:02.576+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:02.855 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:02.576+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:03.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:03.223+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:03.223+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:03.223+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:03.223+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:03.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:03.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:03.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:03.587+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:03.570+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:04.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:03.570+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:04.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:03.570+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:04.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:03.570+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:04.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:04.174+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:04.174+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:04.174+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:04.174+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:05.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:04.630+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:05.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:04.630+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:05.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:04.630+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:05.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:04.630+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:04.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:04.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:04.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:04.611+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:05.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:05.160+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:05.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:05.160+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:05.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:05.160+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:05.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:05.160+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:05.875 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:05.659+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:05.875 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:05.659+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:05.875 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:05.659+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:05.876 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:05.659+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:05.876 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:05.615+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:05.876 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:05.615+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:05.876 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:05.615+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:05.876 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:05.616+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:06.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:06.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:06.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:06.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:06.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:06.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:06.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:06.115+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:07.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:06.638+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:07.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:06.638+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:07.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:06.638+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:07.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:06.638+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:06.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:06.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:07.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:06.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:07.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:06.611+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:07.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:07.164+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:07.164+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:07.164+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:07.164+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:08.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:08:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:08:07.731+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 15250 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:08:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:07.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:07.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:07.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:07.642+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:08.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:07.681+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:08.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:07.681+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:08.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:07.681+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:08.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:07.681+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:08.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:08.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:08.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:08.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:08.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:08.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:08.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:08.142+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:08.609+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:08.609+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:08.609+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:08.609+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:09.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:08.704+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:09.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:08.704+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:09.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:08.704+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:09.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:08.705+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:09.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:09.109+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:09.109+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:09.109+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:09.109+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:10.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:09.627+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:09.627+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:09.627+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:09.627+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:10.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:09.704+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:10.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:09.704+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:10.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:09.704+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:10.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:09.704+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:10.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:10.145+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:10.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:10.145+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:10.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:10.145+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:10.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:10.145+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:11.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:10.588+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:10.588+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:10.588+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:11.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:10.588+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:11.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:10.722+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:11.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:10.722+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:11.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:10.722+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:11.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:10.722+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:11.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:11.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:11.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:11.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:11.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:11.129+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:11.601+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:11.601+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:11.601+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:11.601+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:12.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:11.682+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:11.682+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:11.682+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:11.682+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:12.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:12.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:12.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:12.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:12.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:12.112+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:12.756 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:08:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:08:12.733+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 15334 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:08:12.757 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:12.651+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:12.757 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:12.651+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:12.757 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:12.651+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:12.757 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:12.651+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:12.757 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:12.688+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:12.757 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:12.688+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:12.757 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:12.688+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:12.758 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:12.689+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:13.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:13.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:13.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:13.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:13.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:13.089+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:14.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:13.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:13.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:13.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:13.642+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:14.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:13.709+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:14.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:13.709+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:14.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:13.709+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:14.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:13.709+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:14.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:14.110+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:14.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:14.110+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:14.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:14.110+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:14.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:14.110+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:15.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:14.661+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:15.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:14.661+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:15.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:14.661+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:15.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:14.661+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:14.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:14.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:15.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:14.611+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:15.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:14.611+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:15.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:15.063+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:15.063+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:15.063+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:15.063+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:16.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:15.654+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:16.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:15.654+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:15.654+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:15.655+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:16.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:15.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:16.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:15.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:16.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:15.613+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:16.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:15.613+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:16.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:16.081+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:16.081+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:16.081+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:16.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:16.081+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:17.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:16.645+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:16.645+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:16.645+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:16.646+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:16.609+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:16.609+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:16.609+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:17.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:16.609+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:17.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:17.084+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:17.084+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:17.084+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:17.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:17.084+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:17.859 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:08:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:08:17.736+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 15415 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:08:17.860 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:17.587+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:17.860 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:17.587+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:17.860 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:17.587+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:17.860 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:17.587+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:17.860 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:17.649+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:17.860 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:17.649+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:17.860 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:17.649+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:17.861 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:17.649+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:18.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:18.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:18.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:18.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:18.133+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:19.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:18.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:18.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:18.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:18.551+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:19.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:18.645+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:19.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:18.645+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:19.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:18.645+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:19.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:18.645+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:19.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:19.149+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:19.149+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:19.149+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:19.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:19.149+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:20.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:19.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:19.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:19.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:19.529+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:19.646+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:19.646+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:19.646+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:20.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:19.646+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:20.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:20.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:20.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:20.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:20.131+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:20.540+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:20.540+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:20.540+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:20.540+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:21.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:20.620+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:21.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:20.620+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:21.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:20.620+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:21.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:20.620+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:21.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:21.143+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:21.143+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:21.143+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:21.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:21.143+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:21.516+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:21.516+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:21.516+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:21.516+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:22.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:21.574+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:22.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:21.574+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:22.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:21.574+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:22.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:21.574+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:22.481 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:22.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:22.482 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:22.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:22.482 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:22.189+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:22.482 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:22.189+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:22.738 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:22.480+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:22.739 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:22.480+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:22.739 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:22.480+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:22.739 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:22.480+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:22.739 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:22.536+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:22.739 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:22.536+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:22.739 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:22.536+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:22.740 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:22.536+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:23.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:08:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:08:22.737+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 15499 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:08:23.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:23.438+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:23.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:23.438+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:23.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:23.438+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:23.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:23.439+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:23.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:23.222+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:23.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:23.222+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:23.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:23.222+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:23.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:23.222+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:24.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:23.585+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:24.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:23.585+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:24.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:23.585+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:24.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:23.586+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:24.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:24.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:24.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:24.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:24.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:24.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:24.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:24.433+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:24.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:24.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:24.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:24.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:24.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:24.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:24.246+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:25.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:24.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:25.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:24.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:25.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:24.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:25.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:24.577+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:25.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:25.426+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:25.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:25.426+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:25.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:25.426+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:25.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:25.426+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:25.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:25.201+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:25.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:25.201+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:25.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:25.201+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:25.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:25.201+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:25.594+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:25.594+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:25.594+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:25.595+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:26.501 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:26.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:26.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:26.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:26.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:26.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:26.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:26.428+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:26.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:26.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:26.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:26.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:26.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:26.193+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:27.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:26.593+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:27.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:26.593+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:27.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:26.593+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:27.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:26.593+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:27.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:27.153+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:27.153+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:27.153+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:27.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:27.153+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:27.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:27.394+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:27.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:27.394+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:27.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:27.394+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:27.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:27.395+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:27.865 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:08:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:08:27.739+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 15580 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:08:27.865 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:27.607+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:27.865 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:27.607+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:27.865 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:27.607+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:27.866 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:27.607+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:28.391 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:28.137+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:28.391 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:28.137+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:28.392 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:28.137+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:28.392 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:28.137+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:28.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:28.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:28.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:28.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:28.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:28.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:28.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:28.391+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:29.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:28.652+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:29.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:28.652+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:29.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:28.652+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:29.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:28.652+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:29.501 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:29.385+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:29.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:29.385+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:29.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:29.385+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:29.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:29.385+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:29.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:29.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:29.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:29.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:29.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:29.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:29.141+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:29.977 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:29.680+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:29.977 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:29.680+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:29.977 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:29.680+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:29.978 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:29.680+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T21:08:29.979 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2023-12-16T21:08:29.979 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T21:08:29.979 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:08:29.999 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-12-16T21:08:30.000 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:08:30.059 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:08:30.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:30.111+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:30.111+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:30.111+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:30.112+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:30.726 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:30.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:30.726 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:30.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:30.726 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:30.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:30.727 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:30.423+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:31.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:30.726+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:30.726+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:30.726+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:30.726+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:31.407 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:31.083+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:31.407 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:31.083+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:31.407 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:31.083+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:31.407 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:31.083+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:31.694 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:31.406+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:31.694 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:31.406+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:31.694 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:31.406+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:31.694 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:31.406+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:32.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:31.729+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:31.729+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:31.729+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:31.730+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:32.400 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:32.096+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:32.400 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:32.096+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:32.400 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:32.096+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:32.400 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:32.096+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:32.743 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:32.400+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:32.743 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:32.400+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:32.743 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:32.400+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:32.743 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:32.400+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:33.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:08:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:08:32.741+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 15664 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:08:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:32.777+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:32.777+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:32.777+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:32.777+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:33.440 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:33.063+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:33.441 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:33.063+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:33.441 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:33.063+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:33.441 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:33.063+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:33.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:33.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:33.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:33.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:33.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:33.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:33.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:33.440+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:34.099 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:33.767+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:34.099 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:33.767+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:34.099 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:33.767+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:34.099 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:33.767+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:34.472 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:34.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:34.472 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:34.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:34.472 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:34.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:34.472 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:34.098+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:34.719+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:34.719+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:34.719+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:34.719+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:34.471+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:34.471+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:34.471+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:34.471+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:35.433 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:35.126+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:35.433 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:35.126+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:35.434 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:35.126+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:35.434 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:35.126+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:35.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:35.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:35.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:35.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:35.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:35.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:35.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:35.433+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:36.147 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:35.754+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:36.147 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:35.754+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:36.147 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:35.754+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:36.147 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:35.754+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:36.436 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:36.147+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:36.436 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:36.147+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:36.436 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:36.147+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:36.436 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:36.147+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:36.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:36.436+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:36.436+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:36.436+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:36.436+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:37.189 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:36.772+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:37.189 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:36.772+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:37.189 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:36.772+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:37.189 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:36.772+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:37.450 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:37.188+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:37.451 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:37.188+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:37.451 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:37.188+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:37.451 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:37.189+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:37.745 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:37.450+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:37.745 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:37.450+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:37.745 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:37.450+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:37.746 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:37.450+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:38.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:08:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:08:37.745+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 15745 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:08:38.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:37.794+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:38.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:37.794+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:37.794+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:37.795+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:38.464 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:38.170+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:38.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:38.170+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:38.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:38.170+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:38.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:38.170+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:38.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:38.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:38.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:38.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:38.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:38.465+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:39.130 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:38.786+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:39.130 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:38.786+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:39.130 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:38.786+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:39.130 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:38.787+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:39.437 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:39.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:39.438 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:39.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:39.438 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:39.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:39.438 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:39.130+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:39.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:39.437+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:39.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:39.437+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:39.437+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:39.437+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:40.137 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:39.826+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:40.137 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:39.826+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:40.138 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:39.826+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:40.138 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:39.827+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:40.485 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:40.137+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:40.486 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:40.137+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:40.486 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:40.137+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:40.486 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:40.137+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:40.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:40.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:40.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:40.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:40.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:40.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:40.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:40.485+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:41.139 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:40.785+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:41.139 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:40.785+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:41.140 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:40.785+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:41.140 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:40.785+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:41.439 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:41.139+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:41.439 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:41.139+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:41.440 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:41.139+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:41.440 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:41.139+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:41.694 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:41.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:41.695 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:41.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:41.695 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:41.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:41.695 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:41.439+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:42.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:41.788+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:41.788+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:41.788+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:41.789+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:42.473 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:42.091+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:42.473 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:42.091+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:42.473 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:42.091+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:42.473 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:42.091+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:42.739 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:42.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:42.739 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:42.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:42.739 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:42.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:42.740 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:42.473+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:43.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:08:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:08:42.747+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 15829 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:08:43.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:42.769+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:43.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:42.769+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:43.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:42.769+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:43.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:42.769+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:43.491 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:43.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:43.491 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:43.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:43.491 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:43.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:43.491 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:43.131+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:43.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:43.490+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:43.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:43.490+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:43.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:43.490+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:43.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:43.491+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:44.112 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:43.772+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:44.112 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:43.772+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:44.113 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:43.772+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:44.113 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:43.772+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:44.452 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:44.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:44.452 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:44.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:44.452 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:44.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:44.452 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:44.112+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:44.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:44.451+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:44.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:44.451+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:44.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:44.451+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:44.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:44.452+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:45.124 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:44.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:45.124 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:44.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:45.125 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:44.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:45.125 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:44.801+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:45.453 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:45.124+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:45.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:45.124+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:45.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:45.124+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:45.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:45.124+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:45.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:45.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:45.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:45.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:45.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:45.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:45.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:45.454+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:46.126 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:45.821+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:46.126 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:45.821+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:46.126 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:45.821+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:46.127 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:45.821+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:46.492 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:46.126+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:46.492 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:46.126+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:46.492 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:46.126+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:46.492 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:46.126+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:46.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:46.492+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:46.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:46.492+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:46.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:46.492+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:46.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:46.492+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:47.159 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:46.822+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:47.159 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:46.822+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:47.160 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:46.822+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:47.160 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:46.822+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:47.484 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:47.159+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:47.484 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:47.159+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:47.484 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:47.159+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:47.484 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:47.159+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:47.750 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:47.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:47.750 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:47.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:47.750 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:47.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:47.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:47.484+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:48.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:08:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:08:47.749+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 15910 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:08:48.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:47.872+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:48.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:47.872+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:48.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:47.872+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:48.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:47.872+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:48.463 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:48.162+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:48.464 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:48.162+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:48.464 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:48.162+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:48.464 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:48.162+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:48.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:48.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:48.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:48.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:48.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:48.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:48.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:48.464+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:49.192 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:48.906+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:49.193 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:48.906+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:49.193 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:48.906+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:49.193 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:48.907+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:49.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:49.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:49.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:49.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:49.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:49.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:49.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:49.192+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:49.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:49.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:49.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:49.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:49.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:49.439+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:50.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:49.929+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:50.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:49.929+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:50.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:49.929+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:50.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:49.930+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:50.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:50.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:50.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:50.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:50.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:50.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:50.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:50.152+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:50.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:50.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:50.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:50.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:50.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:50.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:50.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:50.463+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:51.186 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:50.895+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:51.186 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:50.895+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:51.187 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:50.895+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:51.187 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:50.895+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:51.501 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:51.422+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:51.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:51.422+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:51.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:51.422+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:51.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:51.423+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:51.186+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:51.186+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:51.186+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:51.186+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:51.955 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:51.922+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:51.955 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:51.922+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:51.956 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:51.922+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:51.956 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:51.922+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:52.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:52.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:52.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:52.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:52.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:52.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:52.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:52.180+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:52.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:52.449+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:52.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:52.449+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:52.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:52.449+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:52.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:52.449+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:53.192 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:08:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:08:52.752+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 15994 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:08:53.193 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:52.968+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:53.193 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:52.968+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:53.193 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:52.968+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:53.193 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:52.969+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:53.501 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:53.441+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:53.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:53.441+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:53.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:53.441+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:53.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:53.441+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:53.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:53.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:53.192+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:53.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:53.192+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:54.485 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:54.017+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:54.485 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:54.017+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:54.485 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:54.017+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:54.485 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:54.017+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:54.486 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:54.213+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:54.486 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:54.213+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:54.486 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:54.213+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:54.486 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:54.213+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:54.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:54.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:54.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:54.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:54.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:54.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:54.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:54.485+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:55.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:55.219+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:55.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:55.219+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:55.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:55.219+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:55.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:55.220+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:55.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:54.980+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:55.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:54.980+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:55.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:54.980+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:55.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:54.981+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:55.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:55.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:55.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:55.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:55.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:55.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:55.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:55.454+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:56.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:56.185+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:56.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:56.185+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:56.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:56.185+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:56.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:56.185+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:55.943+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:55.943+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:55.943+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:55.943+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:56.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:56.470+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:56.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:56.470+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:56.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:56.470+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:56.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:56.470+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:57.216 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:56.897+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:57.216 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:56.897+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:57.217 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:56.897+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:57.217 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:56.898+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:57.471 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:57.216+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:57.472 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:57.216+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:57.472 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:57.216+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:57.472 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:57.216+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:57.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:57.471+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:57.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:57.471+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:57.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:57.471+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:57.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:57.471+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:58.009 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:08:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:08:57.753+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 16075 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:08:58.009 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:57.896+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:58.009 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:57.896+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:58.010 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:57.896+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:58.010 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:57.896+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:58.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:58.208+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:58.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:58.208+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:58.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:58.208+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:58.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:58.208+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:58.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:58.504+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:58.506 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:58.504+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:58.506 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:58.504+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:58.506 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:58.504+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:59.210 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:58.909+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:59.210 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:58.909+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:59.210 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:58.909+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:08:59.210 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:58.910+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:59.480 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:59.210+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:59.480 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:59.210+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:59.480 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:59.210+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:08:59.480 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:08:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:08:59.210+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:08:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:59.479+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:59.479+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:59.479+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:08:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:08:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:08:59.480+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:00.159 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:59.861+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:00.159 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:59.861+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:00.159 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:59.861+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:00.160 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:08:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:08:59.862+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:00.437 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:00.204+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:00.438 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:00.204+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:00.438 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:00.204+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:00.438 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:00.204+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:00.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:00.437+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:00.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:00.437+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:00.437+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:00.438+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:01.155 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:00.822+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:01.156 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:00.822+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:01.156 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:00.822+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:01.156 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:00.823+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:01.417 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:01.155+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:01.418 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:01.155+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:01.418 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:01.155+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:01.418 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:01.155+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:01.696 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:01.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:01.696 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:01.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:01.696 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:01.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:01.696 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:01.418+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:02.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:01.818+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:01.818+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:01.818+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:01.818+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:02.417 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:02.121+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:02.418 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:02.121+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:02.418 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:02.121+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:02.418 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:02.121+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:02.708 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:02.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:02.708 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:02.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:02.708 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:02.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:02.708 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:02.417+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:03.121 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:09:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:09:02.756+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 16159 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:09:03.121 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:02.814+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:03.121 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:02.814+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:03.121 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:02.814+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:03.122 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:02.815+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:03.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:03.370+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:03.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:03.370+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:03.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:03.370+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:03.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:03.371+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:03.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:03.121+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:03.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:03.121+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:03.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:03.121+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:03.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:03.121+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:03.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:03.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:03.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:03.801+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:04.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:04.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:04.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:04.113+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:04.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:04.359+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:04.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:04.359+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:04.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:04.359+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:04.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:04.359+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:05.069 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:04.781+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:05.070 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:04.781+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:05.070 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:04.781+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:05.070 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:04.781+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:05.347 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:05.070+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:05.347 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:05.070+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:05.347 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:05.070+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:05.348 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:05.070+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:05.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:05.346+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:05.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:05.346+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:05.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:05.346+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:05.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:05.346+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:06.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:05.813+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:06.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:05.813+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:06.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:05.813+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:06.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:05.813+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:06.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:06.058+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:06.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:06.058+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:06.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:06.058+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:06.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:06.058+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:06.378+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:06.378+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:06.378+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:06.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:06.379+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:07.057 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:06.782+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:07.057 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:06.782+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:07.057 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:06.782+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:07.057 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:06.782+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:07.316 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:07.057+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:07.316 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:07.057+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:07.316 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:07.057+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:07.316 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:07.057+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:07.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:07.392+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:07.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:07.392+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:07.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:07.392+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:07.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:07.392+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:08.075 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:09:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:09:07.759+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 16240 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:09:08.075 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:07.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:08.075 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:07.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:08.076 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:07.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:08.076 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:07.801+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:08.425 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:08.075+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:08.425 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:08.075+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:08.425 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:08.075+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:08.425 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:08.075+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:08.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:08.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:08.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:08.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:08.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:08.425+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:09.065 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:08.757+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:09.066 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:08.757+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:09.066 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:08.757+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:09.066 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:08.758+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:09.458 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:09.065+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:09.458 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:09.065+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:09.458 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:09.065+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:09.458 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:09.065+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:09.458+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:09.458+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:09.458+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:09.458+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:10.159 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:09.807+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:10.159 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:09.807+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:10.159 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:09.807+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:10.159 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:09.808+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:10.160 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:10.037+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:10.160 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:10.037+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:10.160 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:10.037+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:10.160 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:10.037+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:10.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:10.446+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:10.446+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:10.446+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:10.447+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:11.067 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:10.780+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:11.067 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:10.780+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:11.068 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:10.780+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:11.068 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:10.780+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:11.453 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:11.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:11.453 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:11.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:11.453 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:11.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:11.453 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:11.067+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:11.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:11.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:11.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:11.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:11.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:11.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:11.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:11.453+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:12.095 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:11.782+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:12.095 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:11.782+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:12.095 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:11.782+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:12.096 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:11.782+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:12.494 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:12.095+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:12.494 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:12.095+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:12.494 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:12.095+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:12.495 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:12.095+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:12.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:12.494+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:12.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:12.494+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:12.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:12.494+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:12.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:12.494+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:13.076 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:09:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:09:12.761+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 16324 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:09:13.077 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:12.785+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:13.077 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:12.785+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:13.077 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:12.785+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:13.077 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:12.785+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:13.448 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:13.076+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:13.449 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:13.076+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:13.449 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:13.076+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:13.449 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:13.076+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:13.448+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:13.448+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:13.448+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:13.449+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:14.044 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:13.790+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:14.045 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:13.790+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:14.045 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:13.790+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:14.045 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:13.791+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:14.404 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:14.045+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:14.405 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:14.045+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:14.405 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:14.045+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:14.405 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:14.045+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:14.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:14.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:14.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:14.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:14.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:14.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:14.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:14.405+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:15.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:15.000+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:15.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:15.000+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:15.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:15.000+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:15.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:15.000+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:14.790+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:14.790+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:14.790+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:14.790+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:15.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:15.367+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:15.367+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:15.367+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:15.367+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:16.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:15.995+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:16.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:15.995+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:16.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:15.995+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:16.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:15.995+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:15.828+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:15.828+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:15.828+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:15.828+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:16.343+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:16.343+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:16.343+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:16.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:16.343+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:17.013 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:16.790+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:17.013 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:16.790+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:17.013 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:16.790+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:17.013 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:16.791+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:17.339 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:17.012+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:17.339 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:17.012+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:17.339 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:17.012+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:17.339 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:17.013+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:17.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:17.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:17.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:17.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:17.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:17.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:17.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:17.339+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:18.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:09:17.764+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 16405 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:09:18.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:17.785+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:18.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:17.785+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:18.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:17.785+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:18.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:17.786+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:18.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:17.963+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:18.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:17.963+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:18.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:17.963+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:18.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:17.964+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:18.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:18.373+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:18.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:18.373+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:18.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:18.373+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:18.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:18.373+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:19.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:18.823+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:19.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:18.823+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:19.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:18.823+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:19.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:18.823+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:19.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:18.956+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:19.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:18.956+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:19.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:18.956+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:19.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:18.956+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:19.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:19.410+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:19.410+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:19.410+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:19.410+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:20.160 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:19.800+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:20.161 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:19.800+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:20.161 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:19.800+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:20.161 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:19.800+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:20.161 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:19.947+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:20.161 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:19.947+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:20.162 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:19.947+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:20.162 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:19.947+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:20.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:20.438+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:20.438+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:20.438+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:20.438+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:21.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:20.837+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:21.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:20.837+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:21.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:20.837+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:21.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:20.837+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:21.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:20.938+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:21.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:20.938+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:21.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:20.938+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:21.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:20.938+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:21.697 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:21.409+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:21.698 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:21.409+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:21.698 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:21.409+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:21.698 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:21.409+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:21.847+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:21.847+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:21.847+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:21.847+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:21.893+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:22.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:21.893+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:22.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:21.893+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:22.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:21.893+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:22.676 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:22.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:22.676 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:22.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:22.676 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:22.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:22.677 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:22.423+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:23.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:09:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:09:22.766+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 16489 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:09:23.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:22.851+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:23.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:22.851+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:23.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:22.851+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:23.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:22.851+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:23.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:22.857+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:23.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:22.857+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:23.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:22.857+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:23.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:22.858+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:23.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:23.424+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:23.424+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:23.424+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:23.424+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:23.847+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:23.847+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:23.847+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:23.847+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:24.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:23.805+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:24.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:23.805+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:24.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:23.805+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:24.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:23.805+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:24.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:24.474+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:24.474+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:24.474+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:24.474+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:25.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:24.860+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:25.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:24.860+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:25.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:24.860+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:25.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:24.861+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:25.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:24.763+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:25.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:24.763+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:25.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:24.763+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:25.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:24.764+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:25.878 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:25.504+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:25.879 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:25.504+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:25.879 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:25.504+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:25.879 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:25.504+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:25.879 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:25.728+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:25.879 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:25.728+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:25.879 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:25.728+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:25.879 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:25.728+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:26.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:25.878+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:26.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:25.878+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:26.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:25.878+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:26.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:25.878+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:26.470+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:26.470+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:26.470+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:26.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:26.470+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:27.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:26.768+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:27.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:26.768+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:27.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:26.768+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:27.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:26.768+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:27.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:26.896+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:27.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:26.896+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:27.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:26.896+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:27.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:26.897+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:27.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:27.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:27.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:27.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:27.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:27.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:27.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:27.452+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:28.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:09:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:09:27.768+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 16570 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:09:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:27.919+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:27.919+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:27.919+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:27.920+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:27.806+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:27.806+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:27.806+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:27.806+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:28.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:28.407+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:28.407+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:28.407+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:28.407+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:29.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:28.945+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:29.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:28.945+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:29.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:28.945+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:29.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:28.945+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:29.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:28.786+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:29.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:28.786+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:29.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:28.786+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:29.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:28.786+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:29.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:29.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:29.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:29.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:29.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:29.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:29.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:29.403+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:30.161 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:29.790+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:30.162 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:29.790+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:30.162 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:29.790+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:30.162 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:29.790+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:30.162 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:29.914+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:30.162 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:29.914+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:30.163 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:29.914+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:30.163 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:29.915+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:30.501 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:30.360+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:30.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:30.360+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:30.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:30.360+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:30.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:30.361+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:31.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:30.797+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:31.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:30.797+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:31.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:30.797+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:31.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:30.797+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:31.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:30.959+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:31.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:30.959+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:31.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:30.959+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:31.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:30.959+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:31.698 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:31.375+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:31.698 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:31.375+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:31.698 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:31.375+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:31.698 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:31.375+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:31.972 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:31.763+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:31.972 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:31.763+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:31.972 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:31.763+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:31.973 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:31.763+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:32.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:31.972+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:32.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:31.972+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:31.972+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:31.973+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:32.736 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:32.336+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:32.736 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:32.336+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:32.737 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:32.336+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:32.737 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:32.336+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:33.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:09:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:09:32.771+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 16654 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:09:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:32.946+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:32.946+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:32.946+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:32.946+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:33.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:32.736+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:33.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:32.736+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:33.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:32.736+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:33.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:32.736+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:33.708 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:33.377+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:33.709 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:33.377+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:33.709 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:33.377+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:33.709 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:33.377+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:33.944+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:33.944+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:33.944+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:33.945+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:34.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:33.708+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:34.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:33.708+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:34.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:33.708+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:34.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:33.709+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:34.715 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:34.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:34.715 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:34.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:34.715 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:34.391+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:34.715 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:34.391+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:34.993 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:34.715+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:34.994 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:34.715+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:34.994 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:34.715+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:34.994 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:34.715+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:35.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:34.994+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:34.994+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:34.994+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:34.994+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:35.673 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:35.407+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:35.674 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:35.407+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:35.674 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:35.407+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:35.674 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:35.407+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:36.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:35.674+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:35.674+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:35.674+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:35.674+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:36.390 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:36.042+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:36.390 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:36.042+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:36.390 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:36.042+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:36.390 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:36.042+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:36.662 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:36.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:36.662 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:36.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:36.662 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:36.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:36.663 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:36.390+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:37.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:36.663+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:37.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:36.663+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:37.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:36.663+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:37.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:36.663+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:37.353 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:37.067+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:37.354 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:37.067+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:37.354 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:37.067+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:37.354 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:37.067+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:37.679 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:37.354+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:37.679 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:37.354+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:37.679 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:37.354+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:37.679 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:37.354+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:38.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:09:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:09:37.774+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 16735 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:09:38.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:37.679+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:38.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:37.679+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:38.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:37.679+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:38.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:37.679+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:38.347 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:38.069+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:38.347 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:38.069+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:38.348 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:38.069+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:38.348 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:38.069+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:38.713 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:38.347+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:38.713 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:38.347+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:38.713 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:38.347+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:38.713 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:38.347+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:39.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:38.713+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:39.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:38.713+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:39.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:38.713+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:39.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:38.713+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:39.354 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:39.091+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:39.354 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:39.091+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:39.354 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:39.091+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:39.355 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:39.091+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:39.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:39.354+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:39.354+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:39.354+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:39.354+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:40.055 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:39.751+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:40.056 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:39.751+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:40.056 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:39.751+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:40.056 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:39.752+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:40.382 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:40.056+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:40.382 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:40.056+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:40.382 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:40.056+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:40.382 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:40.056+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:40.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:40.382+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:40.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:40.382+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:40.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:40.382+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:40.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:40.382+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:41.103 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:40.752+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:41.103 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:40.752+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:41.103 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:40.752+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:41.103 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:40.752+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:41.362 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:41.103+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:41.362 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:41.103+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:41.362 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:41.103+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:41.363 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:41.103+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:41.698 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:41.361+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:41.698 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:41.361+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:41.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:41.361+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:41.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:41.362+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:42.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:41.781+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:42.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:41.781+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:42.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:41.781+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:42.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:41.781+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:42.383 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:42.116+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:42.383 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:42.116+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:42.383 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:42.116+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:42.383 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:42.116+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:42.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:42.382+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:42.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:42.382+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:42.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:42.382+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:42.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:42.382+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:43.126 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:09:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:09:42.775+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 16817 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:09:43.127 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:42.790+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:43.127 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:42.790+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:43.127 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:42.790+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:43.127 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:42.791+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:43.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:43.352+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:43.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:43.352+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:43.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:43.352+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:43.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:43.352+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:43.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:43.126+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:43.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:43.126+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:43.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:43.126+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:43.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:43.126+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:44.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:43.769+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:44.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:43.769+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:44.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:43.769+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:44.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:43.769+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:44.154+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:44.154+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:44.154+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:44.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:44.154+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:44.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:44.358+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:44.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:44.358+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:44.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:44.358+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:44.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:44.358+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:45.166 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:44.768+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:45.166 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:44.768+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:45.166 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:44.768+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:45.166 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:44.768+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:45.166+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:45.166+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:45.166+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:45.166+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:45.320+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:45.320+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:45.320+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:45.320+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:46.185 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:45.774+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:46.188 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:45.774+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:46.189 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:45.774+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:46.189 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:45.774+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:46.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:46.184+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:46.184+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:46.184+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:46.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:46.184+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:46.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:46.273+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:46.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:46.273+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:46.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:46.273+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:46.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:46.273+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:47.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:46.806+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:47.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:46.806+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:47.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:46.806+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:47.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:46.806+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:47.228+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:47.228+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:47.228+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:47.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:47.228+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:47.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:47.291+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:47.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:47.291+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:47.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:47.291+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:47.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:47.292+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:48.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:09:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:09:47.777+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 16899 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:09:48.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:47.769+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:48.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:47.769+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:48.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:47.769+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:48.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:47.769+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:48.274+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:48.274+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:48.274+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:48.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:48.274+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:48.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:48.335+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:48.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:48.335+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:48.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:48.335+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:48.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:48.335+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:49.227 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:48.763+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:49.227 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:48.763+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:49.227 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:48.763+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:49.227 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:48.763+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:49.226+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:49.226+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:49.226+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:49.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:49.226+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:49.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:49.360+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:49.360+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:49.360+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:49.360+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:50.163 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:49.798+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:50.163 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:49.798+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:50.163 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:49.798+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:50.163 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:49.798+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:50.501 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:50.359+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:50.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:50.359+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:50.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:50.359+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:50.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:50.359+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:50.259+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:50.259+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:50.259+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:50.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:50.260+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:51.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:50.798+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:51.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:50.798+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:51.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:50.798+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:51.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:50.798+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:51.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:51.406+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:51.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:51.406+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:51.700 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:51.406+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:51.700 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:51.406+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:51.700 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:51.250+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:51.700 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:51.250+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:51.700 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:51.250+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:51.700 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:51.251+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:52.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:51.791+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:52.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:51.791+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:51.791+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:51.792+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:52.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:52.253+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:52.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:52.253+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:52.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:52.253+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:52.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:52.253+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:52.370+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:52.370+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:52.370+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:52.371+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:53.219 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:09:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:09:52.780+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 16982 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:09:53.219 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:52.752+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:53.219 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:52.752+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:53.220 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:52.752+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:53.220 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:52.752+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:53.219+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:53.219+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:53.219+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:53.219+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:53.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:53.338+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:53.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:53.338+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:53.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:53.338+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:53.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:53.338+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:53.719+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:53.719+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:53.719+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:53.719+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:54.186+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:54.186+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:54.186+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:54.186+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:54.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:54.376+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:54.376+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:54.376+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:54.376+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:55.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:54.674+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:55.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:54.674+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:55.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:54.674+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:55.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:54.674+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:55.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:55.175+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:55.175+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:55.175+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:55.175+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:55.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:55.415+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:55.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:55.415+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:55.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:55.415+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:55.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:55.415+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:55.709+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:55.709+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:55.709+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:55.709+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:56.458 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:56.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:56.459 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:56.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:56.459 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:56.179+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:56.459 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:56.179+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:56.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:56.458+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:56.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:56.458+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:56.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:56.458+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:56.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:56.458+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:56.690+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:56.690+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:56.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:56.690+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:56.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:56.691+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:57.423 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:57.137+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:57.424 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:57.137+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:57.424 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:57.137+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:57.424 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:57.137+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:57.704 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:57.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:57.704 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:57.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:57.704 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:57.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:57.705 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:57.423+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:58.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:09:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:09:57.783+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 17065 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:09:58.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:57.704+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:58.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:57.704+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:58.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:57.704+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:58.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:57.704+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:58.420 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:58.142+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:58.420 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:58.142+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:58.420 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:58.142+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:58.420 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:58.143+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:58.420+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:58.420+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:58.420+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:58.420+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:58.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:58.664+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:58.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:58.664+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:58.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:58.664+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:58.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:58.665+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:59.468 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:59.161+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:59.468 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:59.161+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:59.469 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:59.161+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:09:59.469 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:09:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:09:59.162+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:59.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:59.468+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:59.468+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:59.468+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:09:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:09:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:09:59.468+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:09:59.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:59.688+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:59.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:59.688+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:59.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:59.688+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:09:59.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:09:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:09:59.689+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:00.452 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:00.112+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:00.452 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:00.112+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:00.453 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:00.112+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:00.453 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:00.113+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:00.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:00.644+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:00.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:00.644+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:00.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:00.644+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:00.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:00.645+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:00.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:00.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:00.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:00.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:00.452+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:01.484 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:01.112+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:01.484 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:01.112+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:01.484 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:01.112+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:01.485 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:01.113+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:01.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:01.667+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:01.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:01.667+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:01.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:01.667+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:01.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:01.668+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:01.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:01.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:01.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:01.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:01.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:01.484+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:01.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:01.484+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:02.464 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:02.155+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:02.465 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:02.155+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:02.465 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:02.155+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:02.465 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:02.155+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:02.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:02.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:02.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:02.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:02.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:02.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:02.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:02.465+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:02.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:02.653+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:02.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:02.653+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:02.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:02.653+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:02.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:02.653+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:03.205 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:10:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:10:02.786+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 17148 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:10:03.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:03.205+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:03.205+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:03.205+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:03.205+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:04.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:03.507+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:03.507+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:03.507+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:03.507+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:04.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:03.647+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:04.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:03.647+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:04.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:03.647+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:04.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:03.647+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:04.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:04.213+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:04.213+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:04.213+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:04.213+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:05.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:04.517+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:04.517+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:04.517+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:04.518+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:05.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:04.681+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:05.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:04.681+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:05.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:04.681+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:05.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:04.681+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:05.233+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:05.233+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:05.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:05.233+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:05.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:05.234+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:06.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:05.565+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:05.565+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:05.565+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:05.565+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:06.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:05.654+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:06.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:05.654+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:06.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:05.654+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:06.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:05.654+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:06.526 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:06.265+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:06.527 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:06.265+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:06.527 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:06.265+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:06.527 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:06.265+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:06.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:06.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:06.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:06.527+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:07.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:06.648+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:07.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:06.648+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:07.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:06.648+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:07.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:06.648+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:07.567 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:07.307+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:07.567 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:07.307+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:07.567 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:07.307+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:07.568 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:07.308+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:07.914 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:10:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:10:07.789+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 17230 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:10:07.915 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:07.612+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:07.915 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:07.612+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:07.915 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:07.612+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:07.915 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:07.613+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:07.915 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:07.567+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:07.916 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:07.567+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:07.916 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:07.567+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:07.916 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:07.567+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:08.589 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:08.263+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:08.589 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:08.263+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:08.589 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:08.263+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:08.590 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:08.263+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:09.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:08.589+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:08.589+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:08.589+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:08.589+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:09.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:08.605+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:09.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:08.605+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:09.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:08.605+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:09.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:08.605+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:09.541 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:09.265+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:09.542 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:09.265+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:09.542 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:09.265+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:09.542 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:09.266+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:09.541+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:09.541+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:09.541+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:10.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:09.542+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:10.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:09.630+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:10.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:09.630+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:10.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:09.630+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:10.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:09.631+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:10.500 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:10.234+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:10.500 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:10.234+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:10.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:10.234+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:10.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:10.235+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:10.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:10.500+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:10.500+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:10.500+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:10.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:10.500+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:10.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:10.644+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:10.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:10.644+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:10.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:10.644+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:10.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:10.644+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:11.684 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:11.263+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:11.684 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:11.263+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:11.685 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:11.263+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:11.685 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:11.264+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:11.685 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:11.495+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:11.685 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:11.495+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:11.685 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:11.495+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:11.685 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:11.495+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:11.684+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:11.684+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:11.684+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:11.684+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:12.669 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:12.504+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:12.669 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:12.504+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:12.669 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:12.504+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:12.669 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:12.504+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:12.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:12.294+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:12.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:12.294+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:12.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:12.294+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:12.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:12.295+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:12.919 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:10:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:10:12.792+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 17313 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:10:12.919 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:12.668+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:12.919 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:12.668+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:12.919 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:12.668+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:12.920 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:12.669+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:13.595 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:13.492+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:13.595 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:13.492+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:13.595 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:13.492+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:13.595 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:13.493+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:13.596 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:13.278+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:13.596 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:13.278+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:13.596 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:13.278+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:13.596 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:13.278+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:14.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:13.675+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:14.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:13.675+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:14.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:13.675+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:14.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:13.676+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:14.230+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:14.230+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:14.230+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:14.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:14.231+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:14.899 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:14.671+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:14.900 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:14.671+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:14.900 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:14.671+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:14.900 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:14.672+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:14.901 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:14.503+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:14.901 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:14.503+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:14.901 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:14.503+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:14.902 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:14.504+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:15.549 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:15.264+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:15.549 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:15.264+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:15.549 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:15.264+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:15.549 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:15.264+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:16.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:15.548+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:15.548+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:15.548+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:15.549+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:15.657+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:15.657+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:15.657+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:16.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:15.657+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:16.582 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:16.252+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:16.582 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:16.252+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:16.582 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:16.252+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:16.583 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:16.252+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:16.582+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:16.582+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:16.582+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:16.582+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:17.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:16.634+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:17.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:16.634+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:17.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:16.634+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:17.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:16.634+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:17.231+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:17.231+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:17.231+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:17.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:17.232+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:17.921 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:10:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:10:17.794+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 17395 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:10:17.922 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:17.613+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:17.922 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:17.613+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:17.922 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:17.613+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:17.922 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:17.613+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:17.922 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:17.592+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:17.923 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:17.592+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:17.923 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:17.592+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:17.923 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:17.592+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:18.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:18.202+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:18.202+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:18.202+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:18.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:18.202+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:19.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:18.587+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:18.587+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:18.587+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:18.588+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:18.575+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:18.575+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:18.575+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:18.575+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:19.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:19.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:19.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:19.217+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:19.217+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:20.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:19.591+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:20.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:19.591+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:20.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:19.591+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:20.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:19.592+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:20.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:19.610+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:20.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:19.610+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:20.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:19.610+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:20.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:19.610+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:20.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:20.227+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:20.227+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:20.227+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:20.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:20.228+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:21.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:20.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:20.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:20.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:20.594+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:20.635+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:21.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:20.635+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:21.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:20.635+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:21.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:20.635+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:21.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:21.224+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:21.224+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:21.224+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:21.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:21.225+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:21.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:21.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:21.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:21.587+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:22.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:21.657+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:22.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:21.657+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:22.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:21.657+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:22.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:21.657+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:22.188+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:22.188+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:22.188+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:22.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:22.188+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:22.922 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:10:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:10:22.797+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 17478 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:10:22.922 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:22.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:22.923 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:22.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:22.923 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:22.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:22.923 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:22.633+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:22.923 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:22.634+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:22.923 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:22.635+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:22.923 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:22.635+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:22.923 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:22.635+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:23.496 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:23.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:23.496 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:23.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:23.497 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:23.198+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:23.497 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:23.198+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:23.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:23.645+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:23.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:23.645+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:23.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:23.645+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:23.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:23.645+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:23.661+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:23.661+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:23.661+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:23.662+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:24.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:24.194+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:24.194+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:24.194+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:24.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:24.195+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:25.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:24.654+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:24.654+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:24.654+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:24.654+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:24.678+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:24.678+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:25.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:24.678+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:25.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:24.678+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:25.176+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:25.176+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:25.176+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:25.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:25.176+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:25.649+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:25.649+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:25.649+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:26.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:25.649+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:26.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:25.651+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:26.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:25.651+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:26.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:25.651+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:26.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:25.651+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:26.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:26.138+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:26.138+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:26.138+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:26.138+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:27.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:26.609+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:27.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:26.609+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:27.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:26.609+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:27.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:26.609+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:26.637+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:27.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:26.637+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:27.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:26.637+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:27.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:26.637+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:27.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:27.122+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:27.122+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:27.122+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:27.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:27.122+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:27.926 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:10:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:10:27.800+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 17560 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:10:27.926 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:27.586+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:27.926 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:27.586+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:27.926 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:27.586+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:27.927 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:27.587+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:27.927 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:27.661+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:27.927 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:27.661+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:27.927 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:27.661+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:27.927 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:27.661+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:28.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:28.100+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:28.100+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:28.100+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:28.100+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:29.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:28.622+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:28.622+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:29.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:28.622+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:29.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:28.622+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:29.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:28.611+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:29.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:28.611+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:29.006 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:28.611+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:29.006 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:28.611+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:29.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:29.102+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:29.102+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:29.102+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:29.102+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:30.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:29.578+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:29.578+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:29.578+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:29.578+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:30.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:29.652+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:30.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:29.653+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:30.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:29.653+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:30.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:29.653+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:30.141 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T21:10:30.141 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2023-12-16T21:10:30.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:10:30.142 DEBUG:teuthology.orchestra.run.smithi052:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2023-12-16T21:10:30.202 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/commands/tpm-tools/tpm/tpm_version: Cannot mkdir: Input/output error 2023-12-16T21:10:30.202 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/commands/tpm-tools/tpm/tpm_version/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.203 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/commands/tpm-tools/tpm/tpm_version/00_Descriptions.txt: Cannot open: Input/output error 2023-12-16T21:10:30.203 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/commands/tpm-tools/tpm/tpm_version/tpm_version_tests.sh: Cannot open: Input/output error 2023-12-16T21:10:30.203 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/commands/tpm-tools/tpm/tpm_changeauth: Cannot mkdir: Input/output error 2023-12-16T21:10:30.203 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/commands/tpm-tools/tpm/tpm_changeauth/tpm_changeauth_tests_exp02.sh: Cannot open: Input/output error 2023-12-16T21:10:30.203 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/commands/tpm-tools/tpm/tpm_changeauth/tpm_changeauth_tests.sh: Cannot open: Input/output error 2023-12-16T21:10:30.203 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/commands/tpm-tools/tpm/tpm_changeauth/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.203 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/commands/tpm-tools/tpm/tpm_changeauth/00_Descriptions.txt: Cannot open: Input/output error 2023-12-16T21:10:30.203 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/commands/tpm-tools/tpm/tpm_changeauth/tpm_changeauth_tests_exp01.sh: Cannot open: Input/output error 2023-12-16T21:10:30.203 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/commands/tpm-tools/tpm/tpm_changeauth/tpm_changeauth_tests_exp03.sh: Cannot open: Input/output error 2023-12-16T21:10:30.204 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/commands/tpm-tools/tpm: Cannot utime: Input/output error 2023-12-16T21:10:30.204 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/commands/tpm-tools: Cannot utime: Input/output error 2023-12-16T21:10:30.204 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/commands: Cannot utime: Input/output error 2023-12-16T21:10:30.204 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc: Cannot mkdir: Input/output error 2023-12-16T21:10:30.204 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math: Cannot mkdir: Input/output error 2023-12-16T21:10:30.204 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/atof: Cannot mkdir: Input/output error 2023-12-16T21:10:30.204 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/atof/atof01.c: Cannot open: Input/output error 2023-12-16T21:10:30.204 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/atof/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.204 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float: Cannot mkdir: Input/output error 2023-12-16T21:10:30.205 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/float_iperb.c: Cannot open: Input/output error 2023-12-16T21:10:30.205 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/tfloat.h: Cannot open: Input/output error 2023-12-16T21:10:30.205 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/iperb: Cannot mkdir: Input/output error 2023-12-16T21:10:30.205 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/iperb/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.205 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/iperb/geniperb.c: Cannot open: Input/output error 2023-12-16T21:10:30.205 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/iperb/README: Cannot open: Input/output error 2023-12-16T21:10:30.206 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/iperb/gencosh.c: Cannot open: Input/output error 2023-12-16T21:10:30.206 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/iperb/gentanh.c: Cannot open: Input/output error 2023-12-16T21:10:30.206 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/iperb/gensinh.c: Cannot open: Input/output error 2023-12-16T21:10:30.206 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.206 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/README: Cannot open: Input/output error 2023-12-16T21:10:30.206 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/thread_code.c: Cannot open: Input/output error 2023-12-16T21:10:30.206 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/power: Cannot mkdir: Input/output error 2023-12-16T21:10:30.207 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/power/genfmod.c: Cannot open: Input/output error 2023-12-16T21:10:30.207 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/power/readme: Cannot open: Input/output error 2023-12-16T21:10:30.207 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/power/genpow.c: Cannot open: Input/output error 2023-12-16T21:10:30.207 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/power/genceil.c: Cannot open: Input/output error 2023-12-16T21:10:30.207 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/power/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.207 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/power/genpower.c: Cannot open: Input/output error 2023-12-16T21:10:30.207 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/power/genfabs.c: Cannot open: Input/output error 2023-12-16T21:10:30.207 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/power/gensqrt.c: Cannot open: Input/output error 2023-12-16T21:10:30.207 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/power/genfloor.c: Cannot open: Input/output error 2023-12-16T21:10:30.208 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/float_trigo.c: Cannot open: Input/output error 2023-12-16T21:10:30.208 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/float_exp_log.c: Cannot open: Input/output error 2023-12-16T21:10:30.208 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/bessel: Cannot mkdir: Input/output error 2023-12-16T21:10:30.208 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/bessel/readme: Cannot open: Input/output error 2023-12-16T21:10:30.208 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/bessel/genj0.c: Cannot open: Input/output error 2023-12-16T21:10:30.208 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/bessel/geny1.c: Cannot open: Input/output error 2023-12-16T21:10:30.208 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/bessel/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.208 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/bessel/genj1.c: Cannot open: Input/output error 2023-12-16T21:10:30.208 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/bessel/genlgamma.c: Cannot open: Input/output error 2023-12-16T21:10:30.208 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/bessel/genbessel.c: Cannot open: Input/output error 2023-12-16T21:10:30.209 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/bessel/geny0.c: Cannot open: Input/output error 2023-12-16T21:10:30.209 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/trigo: Cannot mkdir: Input/output error 2023-12-16T21:10:30.209 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/trigo/gentan.c: Cannot open: Input/output error 2023-12-16T21:10:30.209 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/trigo/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.209 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/trigo/genacos.c: Cannot open: Input/output error 2023-12-16T21:10:30.209 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/trigo/genatan.c: Cannot open: Input/output error 2023-12-16T21:10:30.209 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/trigo/genasin.c: Cannot open: Input/output error 2023-12-16T21:10:30.209 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/trigo/README: Cannot open: Input/output error 2023-12-16T21:10:30.210 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/trigo/genatan2.c: Cannot open: Input/output error 2023-12-16T21:10:30.210 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/trigo/gencos.c: Cannot open: Input/output error 2023-12-16T21:10:30.210 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/trigo/gentrigo.c: Cannot open: Input/output error 2023-12-16T21:10:30.210 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/trigo/gensin.c: Cannot open: Input/output error 2023-12-16T21:10:30.210 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/main.c: Cannot open: Input/output error 2023-12-16T21:10:30.210 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/float_bessel.c: Cannot open: Input/output error 2023-12-16T21:10:30.210 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/float_power.c: Cannot open: Input/output error 2023-12-16T21:10:30.210 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/exp_log: Cannot mkdir: Input/output error 2023-12-16T21:10:30.211 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/exp_log/readme: Cannot open: Input/output error 2023-12-16T21:10:30.211 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/exp_log/genfrexp.c: Cannot open: Input/output error 2023-12-16T21:10:30.211 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/exp_log/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.211 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/exp_log/genhypot.c: Cannot open: Input/output error 2023-12-16T21:10:30.211 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/exp_log/genexp.c: Cannot open: Input/output error 2023-12-16T21:10:30.211 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/exp_log/genldexp.c: Cannot open: Input/output error 2023-12-16T21:10:30.211 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/exp_log/genlog10.c: Cannot open: Input/output error 2023-12-16T21:10:30.211 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/exp_log/genmodf.c: Cannot open: Input/output error 2023-12-16T21:10:30.211 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/exp_log/genexp_log.c: Cannot open: Input/output error 2023-12-16T21:10:30.212 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/float/exp_log/genlog.c: Cannot open: Input/output error 2023-12-16T21:10:30.212 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.212 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/abs: Cannot mkdir: Input/output error 2023-12-16T21:10:30.212 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/abs/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.212 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/abs/abs01.c: Cannot open: Input/output error 2023-12-16T21:10:30.212 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/fptests: Cannot mkdir: Input/output error 2023-12-16T21:10:30.212 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/fptests/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.212 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/fptests/fptest01.c: Cannot open: Input/output error 2023-12-16T21:10:30.212 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/fptests/fptest02.c: Cannot open: Input/output error 2023-12-16T21:10:30.213 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/nextafter: Cannot mkdir: Input/output error 2023-12-16T21:10:30.213 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/nextafter/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.213 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/math/nextafter/nextafter01.c: Cannot open: Input/output error 2023-12-16T21:10:30.213 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/crash: Cannot mkdir: Input/output error 2023-12-16T21:10:30.213 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/crash/crash01.c: Cannot open: Input/output error 2023-12-16T21:10:30.213 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/crash/crash02.c: Cannot open: Input/output error 2023-12-16T21:10:30.213 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/crash/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.213 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.213 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/tcore_patch_test_suites: Cannot mkdir: Input/output error 2023-12-16T21:10:30.214 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/tcore_patch_test_suites/tcore.exp: Cannot open: Input/output error 2023-12-16T21:10:30.214 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/tcore_patch_test_suites/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.214 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/tcore_patch_test_suites/tcore.sh: Cannot open: Input/output error 2023-12-16T21:10:30.214 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/tcore_patch_test_suites/tcore.c: Cannot open: Input/output error 2023-12-16T21:10:30.214 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/f00f: Cannot mkdir: Input/output error 2023-12-16T21:10:30.214 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/f00f/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.214 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/misc/f00f/f00f.c: Cannot open: Input/output error 2023-12-16T21:10:30.214 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network: Cannot mkdir: Input/output error 2023-12-16T21:10:30.214 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds: Cannot mkdir: Input/output error 2023-12-16T21:10:30.215 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ping: Cannot mkdir: Input/output error 2023-12-16T21:10:30.215 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ping/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.215 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ping/ping01: Cannot open: Input/output error 2023-12-16T21:10:30.215 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/echo: Cannot mkdir: Input/output error 2023-12-16T21:10:30.215 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/echo/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.215 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/echo/createfile.c: Cannot open: Input/output error 2023-12-16T21:10:30.215 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/echo/echo01: Cannot open: Input/output error 2023-12-16T21:10:30.215 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/echo/echoes.c: Cannot open: Input/output error 2023-12-16T21:10:30.215 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/telnet: Cannot mkdir: Input/output error 2023-12-16T21:10:30.216 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/telnet/telnet01: Cannot open: Input/output error 2023-12-16T21:10:30.216 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/telnet/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.216 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/rsh: Cannot mkdir: Input/output error 2023-12-16T21:10:30.216 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/rsh/rsh01: Cannot open: Input/output error 2023-12-16T21:10:30.216 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/rsh/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.216 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/netstat: Cannot mkdir: Input/output error 2023-12-16T21:10:30.216 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/netstat/netstat01: Cannot open: Input/output error 2023-12-16T21:10:30.216 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/netstat/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.216 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.216 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/tcpdump: Cannot mkdir: Input/output error 2023-12-16T21:10:30.217 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/tcpdump/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.217 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/tcpdump/tcpdump01: Cannot open: Input/output error 2023-12-16T21:10:30.217 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/finger: Cannot mkdir: Input/output error 2023-12-16T21:10:30.217 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/finger/finger01: Cannot open: Input/output error 2023-12-16T21:10:30.217 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/finger/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.217 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ftp: Cannot mkdir: Input/output error 2023-12-16T21:10:30.217 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ftp/ftp02_s1: Cannot open: Input/output error 2023-12-16T21:10:30.217 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ftp/ftp05: Cannot open: Input/output error 2023-12-16T21:10:30.217 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ftp/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.218 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ftp/ftp_setup_vsftp_conf: Cannot open: Input/output error 2023-12-16T21:10:30.218 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ftp/ftp02: Cannot open: Input/output error 2023-12-16T21:10:30.218 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ftp/ftp04: Cannot open: Input/output error 2023-12-16T21:10:30.218 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ftp/ftp01: Cannot open: Input/output error 2023-12-16T21:10:30.218 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ftp/ftp_set_passwd: Cannot open: Input/output error 2023-12-16T21:10:30.218 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ftp/ftp03: Cannot open: Input/output error 2023-12-16T21:10:30.218 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/README: Cannot open: Input/output error 2023-12-16T21:10:30.218 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/host: Cannot mkdir: Input/output error 2023-12-16T21:10:30.218 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/host/host01: Cannot open: Input/output error 2023-12-16T21:10:30.218 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/host/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.219 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/rlogin: Cannot mkdir: Input/output error 2023-12-16T21:10:30.219 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/rlogin/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.219 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/rlogin/rlogin01: Cannot open: Input/output error 2023-12-16T21:10:30.219 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/perf_lan: Cannot mkdir: Input/output error 2023-12-16T21:10:30.219 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/perf_lan/pingpong6.c: Cannot open: Input/output error 2023-12-16T21:10:30.219 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/perf_lan/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.219 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/perf_lan/perf_lan: Cannot open: Input/output error 2023-12-16T21:10:30.219 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/perf_lan/pingpong.c: Cannot open: Input/output error 2023-12-16T21:10:30.220 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/rdist: Cannot mkdir: Input/output error 2023-12-16T21:10:30.220 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/rdist/rdist01: Cannot open: Input/output error 2023-12-16T21:10:30.220 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/rdist/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.220 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/sendfile: Cannot mkdir: Input/output error 2023-12-16T21:10:30.220 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/sendfile/sendfile01: Cannot open: Input/output error 2023-12-16T21:10:30.220 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/sendfile/testsf_s.c: Cannot open: Input/output error 2023-12-16T21:10:30.220 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/sendfile/testsf_c.c: Cannot open: Input/output error 2023-12-16T21:10:30.220 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/sendfile/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.220 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/sendfile/SF_Server: Cannot open: Input/output error 2023-12-16T21:10:30.220 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/arp: Cannot mkdir: Input/output error 2023-12-16T21:10:30.221 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/arp/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.221 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/arp/arp01: Cannot open: Input/output error 2023-12-16T21:10:30.221 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/rcp: Cannot mkdir: Input/output error 2023-12-16T21:10:30.221 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/rcp/rcp01: Cannot open: Input/output error 2023-12-16T21:10:30.221 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/rcp/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.221 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/rwho: Cannot mkdir: Input/output error 2023-12-16T21:10:30.221 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/rwho/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.221 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/rwho/rwho01: Cannot open: Input/output error 2023-12-16T21:10:30.221 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/include: Cannot mkdir: Input/output error 2023-12-16T21:10:30.222 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/include/netdefs.h: Cannot open: Input/output error 2023-12-16T21:10:30.222 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/Makefile.inc: Cannot open: Input/output error 2023-12-16T21:10:30.222 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ssh: Cannot mkdir: Input/output error 2023-12-16T21:10:30.222 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ssh/ssh01_s1: Cannot open: Input/output error 2023-12-16T21:10:30.222 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ssh/ssh_set_passwd: Cannot open: Input/output error 2023-12-16T21:10:30.222 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ssh/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.222 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ssh/ssh02: Cannot open: Input/output error 2023-12-16T21:10:30.222 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ssh/ssh01: Cannot open: Input/output error 2023-12-16T21:10:30.222 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ssh/ssh02_s1: Cannot open: Input/output error 2023-12-16T21:10:30.223 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ssh/ssh03: Cannot open: Input/output error 2023-12-16T21:10:30.223 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/tcp_cmds/ssh/ssh03_s1: Cannot open: Input/output error 2023-12-16T21:10:30.223 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast: Cannot mkdir: Input/output error 2023-12-16T21:10:30.223 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.223 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_cmds: Cannot mkdir: Input/output error 2023-12-16T21:10:30.223 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_cmds/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.223 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_cmds/mc_cmds: Cannot open: Input/output error 2023-12-16T21:10:30.223 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_opts: Cannot mkdir: Input/output error 2023-12-16T21:10:30.223 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_opts/opts_e.c: Cannot open: Input/output error 2023-12-16T21:10:30.224 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_opts/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.224 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_opts/mc_opts: Cannot open: Input/output error 2023-12-16T21:10:30.224 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_opts/opts.c: Cannot open: Input/output error 2023-12-16T21:10:30.224 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/README: Cannot open: Input/output error 2023-12-16T21:10:30.224 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_member: Cannot mkdir: Input/output error 2023-12-16T21:10:30.224 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_member/TooManyGroups: Cannot open: Input/output error 2023-12-16T21:10:30.224 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_member/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.224 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_member/member.c: Cannot open: Input/output error 2023-12-16T21:10:30.224 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_member/ManyGroups: Cannot open: Input/output error 2023-12-16T21:10:30.224 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_member/mc_member: Cannot open: Input/output error 2023-12-16T21:10:30.225 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_commo: Cannot mkdir: Input/output error 2023-12-16T21:10:30.225 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_commo/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.225 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_commo/README.1st: Cannot open: Input/output error 2023-12-16T21:10:30.225 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_commo/mc_recv.c: Cannot open: Input/output error 2023-12-16T21:10:30.225 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_commo/mc_commo: Cannot open: Input/output error 2023-12-16T21:10:30.225 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/multicast/mc_commo/mc_send.c: Cannot open: Input/output error 2023-12-16T21:10:30.225 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can: Cannot mkdir: Input/output error 2023-12-16T21:10:30.225 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests: Cannot mkdir: Input/output error 2023-12-16T21:10:30.226 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-raw-sendto.c: Cannot open: Input/output error 2023-12-16T21:10:30.226 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-err.c: Cannot open: Input/output error 2023-12-16T21:10:30.226 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-packet.c: Cannot open: Input/output error 2023-12-16T21:10:30.226 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-bcm-filter.c: Cannot open: Input/output error 2023-12-16T21:10:30.226 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-bcm-dump.c: Cannot open: Input/output error 2023-12-16T21:10:30.226 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/canecho.c: Cannot open: Input/output error 2023-12-16T21:10:30.226 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-bcm-rx-sendto.c: Cannot open: Input/output error 2023-12-16T21:10:30.226 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-bcm-single.c: Cannot open: Input/output error 2023-12-16T21:10:30.226 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-bcm-server.c: Cannot open: Input/output error 2023-12-16T21:10:30.226 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-bcm-tx-sendto.c: Cannot open: Input/output error 2023-12-16T21:10:30.227 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-filter-server.c: Cannot open: Input/output error 2023-12-16T21:10:30.227 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.227 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-bcm-rtr.c: Cannot open: Input/output error 2023-12-16T21:10:30.227 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/00_Descriptions.txt: Cannot open: Input/output error 2023-12-16T21:10:30.227 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-raw.c: Cannot open: Input/output error 2023-12-16T21:10:30.227 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/run_ltp-can_tests.sh: Cannot open: Input/output error 2023-12-16T21:10:30.227 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-bcm-throttle.c: Cannot open: Input/output error 2023-12-16T21:10:30.227 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-bcm-cycle.c: Cannot open: Input/output error 2023-12-16T21:10:30.227 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-filter-master.c: Cannot open: Input/output error 2023-12-16T21:10:30.228 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-proc.c: Cannot open: Input/output error 2023-12-16T21:10:30.228 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-raw-filter.c: Cannot open: Input/output error 2023-12-16T21:10:30.228 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/INSTALL: Cannot open: Input/output error 2023-12-16T21:10:30.228 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/can/filter-tests/tst-bcm-tx_read.c: Cannot open: Input/output error 2023-12-16T21:10:30.228 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/xinetd: Cannot mkdir: Input/output error 2023-12-16T21:10:30.228 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/xinetd/xinetd_tests.sh: Cannot open: Input/output error 2023-12-16T21:10:30.228 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/xinetd/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.228 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.228 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/lib6: Cannot mkdir: Input/output error 2023-12-16T21:10:30.228 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/lib6/in6_02.c: Cannot open: Input/output error 2023-12-16T21:10:30.229 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/lib6/in6_01.c: Cannot open: Input/output error 2023-12-16T21:10:30.229 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/lib6/runcc.c: Cannot open: Input/output error 2023-12-16T21:10:30.229 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/lib6/asapi_01.c: Cannot open: Input/output error 2023-12-16T21:10:30.229 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/lib6/Makefile: Cannot open: Input/output error 2023-12-16T21:10:30.229 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/lib6/asapi_07.c: Cannot open: Input/output error 2023-12-16T21:10:30.229 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/lib6/asapi_02.c: Cannot open: Input/output error 2023-12-16T21:10:30.229 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/lib6/asapi_03.c: Cannot open: Input/output error 2023-12-16T21:10:30.229 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/lib6/asapi_05.c: Cannot open: Input/output error 2023-12-16T21:10:30.229 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases/network/lib6/runcc.h: Cannot open: Input/output error 2023-12-16T21:10:30.230 INFO:tasks.workunit.client.0.smithi052.stderr: 2023-12-16T21:10:30.230 INFO:tasks.workunit.client.0.smithi052.stderr:gzip: stdin: Stale file handle 2023-12-16T21:10:30.230 INFO:tasks.workunit.client.0.smithi052.stderr:tar: Unexpected EOF in archive 2023-12-16T21:10:30.230 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2023-12-16T21:10:30.230 INFO:tasks.workunit.client.0.smithi052.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2023-12-16T21:10:30.230 INFO:tasks.workunit.client.0.smithi052.stderr:tar: Error is not recoverable: exiting now 2023-12-16T21:10:30.231 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-12-16T21:10:30.231 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2023-12-16T21:10:30.231 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-16T21:10:30.232 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi052.front.sepia.ceph.com 2023-12-16T21:10:30.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:10:30.232 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-16T21:10:30.294 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-12-16T21:10:30.295 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:10:30.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:30.061+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:30.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:30.061+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:30.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:30.061+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:30.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:30.062+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:31.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:30.679+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:31.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:30.679+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:31.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:30.679+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:31.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:30.679+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:30.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:30.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:30.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:31.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:30.595+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:31.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:31.018+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:31.018+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:31.018+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:31.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:31.018+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:32.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:31.660+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:32.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:31.660+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:32.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:31.660+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:32.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:31.660+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:32.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:31.622+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:32.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:31.622+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:32.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:31.622+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:32.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:31.622+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:32.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:32.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:32.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:32.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:32.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:32.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:32.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:32.039+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:32.925 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:10:32.802+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 17643 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:10:32.925 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:32.635+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:32.926 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:32.635+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:32.926 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:32.635+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:32.926 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:32.635+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:32.926 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:32.591+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:32.926 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:32.591+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:32.926 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:32.591+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:32.926 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:32.591+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:33.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:32.997+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:33.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:32.997+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:33.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:32.997+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:33.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:32.997+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:33.596 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:33.557+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:33.596 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:33.557+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:33.596 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:33.557+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:33.596 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:33.557+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:34.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:33.663+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:34.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:33.663+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:34.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:33.663+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:34.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:33.664+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:34.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:34.047+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:34.047+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:34.047+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:34.047+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:34.629+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:34.629+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:34.629+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:35.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:34.629+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:35.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:34.601+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:35.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:34.601+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:35.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:34.601+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:35.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:34.601+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:35.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:35.080+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:35.080+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:35.080+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:35.080+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:36.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:35.653+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:35.653+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:35.653+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:35.653+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:35.595+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:35.595+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:36.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:35.595+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:36.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:35.595+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:36.403 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:36.066+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:36.403 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:36.066+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:36.404 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:36.066+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:36.404 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:36.066+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:36.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:36.608+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:36.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:36.608+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:36.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:36.608+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:36.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:36.609+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:36.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:36.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:36.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:36.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:36.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:36.631+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:37.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:37.024+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:37.024+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:37.024+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:37.024+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:37.933 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:10:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:10:37.805+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 17726 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:10:37.934 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:37.615+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:37.934 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:37.615+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:37.934 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:37.615+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:37.934 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:37.615+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:37.935 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:37.672+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:37.935 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:37.672+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:37.935 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:37.672+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:37.935 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:37.672+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:38.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:38.002+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:38.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:38.002+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:38.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:38.002+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:38.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:38.002+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:38.988 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:38.589+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:38.988 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:38.589+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:38.989 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:38.589+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:38.989 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:38.590+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:38.989 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:38.695+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:38.989 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:38.695+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:38.989 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:38.695+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:38.989 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:38.695+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:39.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:38.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:39.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:38.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:39.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:38.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:39.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:38.988+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:39.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:39.693+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:39.693+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:39.693+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:39.693+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:39.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:39.557+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:39.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:39.557+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:39.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:39.557+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:39.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:39.557+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:40.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:39.998+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:40.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:39.998+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:40.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:39.998+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:40.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:39.998+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:40.558+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:40.558+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:40.558+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:40.558+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:40.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:40.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:40.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:40.735+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:41.001+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:41.001+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:41.001+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:41.002+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:42.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:41.698+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:41.698+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:41.698+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:41.698+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:42.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:41.518+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:42.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:41.518+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:42.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:41.518+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:42.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:41.518+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:42.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:42.025+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:42.025+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:42.025+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:42.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:42.025+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:42.808 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:42.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:42.808 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:42.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:42.808 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:42.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:42.808 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:42.728+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:42.808 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:42.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:42.808 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:42.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:42.809 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:42.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:42.809 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:42.508+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:43.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:10:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:10:42.807+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 17810 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:10:43.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:43.015+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:43.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:43.015+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:43.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:43.015+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:43.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:43.015+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:43.983 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:43.758+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:43.983 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:43.758+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:43.983 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:43.758+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:43.983 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:43.758+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:43.983 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:43.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:43.984 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:43.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:43.984 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:43.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:43.984 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:43.509+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:44.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:43.982+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:44.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:43.982+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:44.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:43.982+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:44.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:43.983+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:44.993 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:44.712+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:44.993 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:44.712+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:44.993 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:44.712+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:44.993 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:44.712+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:44.994 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:44.549+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:44.994 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:44.549+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:44.994 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:44.549+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:44.994 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:44.550+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:45.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:44.992+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:45.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:44.992+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:45.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:44.992+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:45.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:44.993+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:46.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:45.524+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:46.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:45.524+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:46.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:45.524+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:46.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:45.524+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:45.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:46.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:45.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:46.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:45.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:46.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:45.665+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:46.488 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:46.019+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:46.488 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:46.019+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:46.488 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:46.019+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:46.488 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:46.019+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:46.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:46.681+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:46.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:46.681+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:46.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:46.681+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:46.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:46.681+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:46.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:46.488+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:46.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:46.488+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:46.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:46.488+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:46.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:46.488+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:47.465 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:47.053+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:47.466 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:47.053+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:47.466 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:47.053+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:47.466 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:47.053+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:47.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:47.709+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:47.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:47.709+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:47.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:47.709+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:47.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:47.710+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:47.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:47.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:47.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:47.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:47.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:47.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:47.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:47.466+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:48.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:10:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:10:47.810+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 17893 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:10:48.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:48.008+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:48.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:48.008+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:48.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:48.008+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:48.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:48.009+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:48.975 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:48.663+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:48.975 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:48.663+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:48.975 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:48.663+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:48.975 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:48.663+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:48.975 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:48.512+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:48.976 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:48.512+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:48.976 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:48.512+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:48.976 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:48.512+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:49.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:48.981+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:49.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:48.981+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:49.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:48.981+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:49.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:48.982+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:49.945 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:49.643+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:49.946 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:49.643+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:49.946 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:49.643+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:49.946 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:49.643+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:49.946 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:49.547+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:49.946 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:49.547+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:49.946 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:49.547+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:49.946 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:49.548+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:50.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:49.945+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:50.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:49.945+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:50.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:49.945+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:50.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:49.945+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:50.988 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:50.672+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:50.989 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:50.672+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:50.989 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:50.672+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:50.989 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:50.672+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:50.989 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:50.545+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:50.990 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:50.545+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:50.990 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:50.545+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:50.990 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:50.545+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:51.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:50.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:51.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:50.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:51.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:50.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:51.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:50.988+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:51.946 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:51.626+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:51.946 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:51.626+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:51.947 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:51.626+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:51.947 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:51.627+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:51.947 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:51.538+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:51.947 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:51.538+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:51.947 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:51.538+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:51.947 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:51.538+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:52.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:51.946+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:52.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:51.946+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:52.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:51.946+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:52.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:51.946+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:52.813 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:52.583+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:52.813 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:52.583+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:52.813 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:52.583+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:52.813 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:52.584+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:52.814 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:52.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:52.814 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:52.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:52.814 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:52.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:52.814 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:52.529+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:53.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:10:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:10:52.812+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 17975 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:10:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:52.950+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:52.950+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:52.950+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:52.950+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:53.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:53.607+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:53.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:53.607+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:53.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:53.607+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:53.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:53.608+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:53.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:53.482+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:53.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:53.482+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:53.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:53.482+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:53.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:53.482+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:54.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:53.980+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:54.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:53.980+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:54.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:53.980+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:54.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:53.980+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:54.835 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:54.626+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:54.835 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:54.626+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:54.836 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:54.626+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:54.836 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:54.626+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:54.836 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:54.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:54.836 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:54.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:54.836 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:54.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:54.837 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:54.508+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:55.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:54.957+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:55.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:54.957+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:55.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:54.957+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:55.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:54.957+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:55.973 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:55.521+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:55.973 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:55.521+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:55.973 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:55.521+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:55.973 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:55.521+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:55.974 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:55.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:55.974 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:55.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:55.974 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:55.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:55.974 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:55.590+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:56.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:55.972+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:55.972+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:55.972+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:55.973+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:56.991 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:56.561+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:56.991 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:56.561+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:56.991 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:56.561+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:56.991 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:56.561+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:56.991 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:56.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:56.992 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:56.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:56.992 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:56.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:56.992 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:56.563+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:57.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:56.991+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:56.991+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:56.991+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:56.991+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:57.826 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:10:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:10:57.815+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 18057 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:10:57.827 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:57.573+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:57.827 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:57.573+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:57.827 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:57.573+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:57.827 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:57.573+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:57.827 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:57.609+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:57.828 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:57.609+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:57.828 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:57.609+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:57.828 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:57.609+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:58.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:57.982+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:58.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:57.982+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:58.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:57.982+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:58.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:57.982+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:58.970 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:58.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:58.970 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:58.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:58.970 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:58.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:58.971 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:58.551+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:58.971 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:58.608+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:58.971 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:58.608+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:58.971 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:58.608+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:58.971 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:58.609+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:59.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:58.970+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:59.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:58.970+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:59.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:58.970+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:10:59.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:58.970+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:59.928 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:59.509+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:59.929 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:59.509+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:59.929 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:59.509+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:10:59.929 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:10:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:10:59.509+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:10:59.929 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:59.638+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:59.929 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:59.638+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:59.929 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:59.638+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:10:59.930 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:10:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:10:59.638+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:00.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:59.929+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:00.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:59.929+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:00.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:59.929+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:00.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:10:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:10:59.929+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:00.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:00.598+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:00.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:00.599+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:00.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:00.599+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:00.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:00.599+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:00.461+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:00.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:00.461+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:00.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:00.461+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:00.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:00.461+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:01.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:00.906+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:01.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:00.906+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:01.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:00.906+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:01.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:00.906+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:01.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:01.575+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:01.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:01.575+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:01.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:01.575+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:01.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:01.575+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:01.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:01.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:01.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:01.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:01.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:01.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:01.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:01.462+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:02.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:01.904+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:02.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:01.904+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:02.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:01.904+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:02.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:01.904+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:02.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:02.501+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:02.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:02.501+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:02.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:02.501+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:02.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:02.501+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:02.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:02.603+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:02.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:02.603+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:02.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:02.603+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:02.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:02.603+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:03.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:11:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:11:02.818+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 18141 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:11:03.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:02.865+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:03.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:02.865+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:03.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:02.865+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:03.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:02.865+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:03.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:03.483+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:03.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:03.483+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:03.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:03.483+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:03.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:03.483+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:03.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:03.578+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:03.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:03.578+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:03.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:03.578+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:03.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:03.578+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:04.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:03.817+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:04.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:03.817+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:03.817+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:03.817+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:04.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:04.627+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:04.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:04.627+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:04.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:04.627+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:04.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:04.627+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:04.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:04.465+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:04.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:04.465+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:04.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:04.465+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:04.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:04.465+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:05.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:04.773+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:04.773+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:04.773+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:04.773+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:05.781 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:05.667+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:05.781 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:05.667+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:05.781 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:05.667+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:05.781 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:05.667+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:05.782 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:05.505+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:05.782 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:05.505+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:05.782 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:05.505+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:05.782 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:05.505+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:06.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:05.779+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:06.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:05.780+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:06.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:05.780+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:06.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:05.780+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:07.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:06.656+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:07.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:06.656+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:07.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:06.656+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:07.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:06.656+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:07.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:06.758+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:07.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:06.758+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:07.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:06.758+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:07.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:06.758+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:07.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:06.524+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:07.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:06.524+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:07.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:06.524+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:07.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:06.524+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:07.617 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:07.501+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:07.617 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:07.501+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:07.617 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:07.501+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:07.617 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:07.501+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:07.946 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:11:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:11:07.819+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 18224 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:11:07.946 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:07.645+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:07.946 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:07.645+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:07.947 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:07.645+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:07.947 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:07.645+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:07.947 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:07.781+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:07.947 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:07.781+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:07.947 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:07.781+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:07.947 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:07.781+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:08.739+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:08.739+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:08.739+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:08.740+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:08.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:08.617+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:08.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:08.617+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:08.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:08.617+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:08.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:08.617+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:08.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:08.496+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:08.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:08.496+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:08.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:08.496+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:08.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:08.497+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:10.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:09.708+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:10.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:09.708+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:10.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:09.708+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:10.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:09.708+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:10.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:09.647+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:10.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:09.647+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:10.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:09.647+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:10.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:09.647+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:10.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:09.519+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:10.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:09.519+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:10.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:09.519+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:10.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:09.519+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:11.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:10.735+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:11.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:10.735+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:11.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:10.735+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:11.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:10.735+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:11.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:10.666+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:11.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:10.666+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:11.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:10.666+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:11.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:10.666+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:11.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:10.559+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:11.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:10.559+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:11.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:10.559+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:11.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:10.559+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:12.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:11.710+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:12.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:11.710+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:12.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:11.710+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:12.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:11.710+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:11.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:11.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:12.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:11.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:12.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:11.585+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:12.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:11.675+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:12.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:11.675+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:12.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:11.675+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:12.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:11.675+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:12.846 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:11:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:11:12.821+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 18308 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:11:12.846 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:12.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:12.846 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:12.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:12.847 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:12.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:12.847 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:12.584+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:12.847 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:12.684+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:12.847 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:12.684+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:12.847 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:12.684+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:12.847 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:12.684+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:12.848 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:12.710+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:12.848 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:12.710+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:12.848 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:12.710+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:12.848 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:12.710+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:14.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:13.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:13.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:13.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:14.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:13.585+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:14.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:13.714+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:14.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:13.714+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:14.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:13.714+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:14.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:13.714+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:14.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:13.662+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:14.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:13.662+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:14.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:13.662+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:14.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:13.662+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:15.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:14.615+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:14.615+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:14.615+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:14.615+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:15.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:14.740+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:15.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:14.740+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:15.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:14.740+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:15.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:14.740+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:15.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:14.647+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:15.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:14.647+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:15.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:14.647+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:15.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:14.647+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:16.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:15.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:15.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:15.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:15.614+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:15.685+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:15.685+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:16.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:15.685+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:16.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:15.685+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:15.714+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:15.714+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:15.714+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:15.714+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:17.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:16.736+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:16.736+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:16.736+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:16.736+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:16.619+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:17.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:16.619+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:17.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:16.619+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:17.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:16.619+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:17.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:16.697+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:17.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:16.697+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:17.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:16.697+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:17.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:16.697+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:17.949 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:11:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:11:17.824+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 18391 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:11:17.949 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:17.694+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:17.950 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:17.694+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:17.950 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:17.694+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:17.950 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:17.694+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:17.950 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:17.759+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:17.950 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:17.759+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:17.950 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:17.759+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:17.950 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:17.760+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:17.951 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:17.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:17.951 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:17.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:17.951 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:17.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:17.951 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:17.631+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:19.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:18.766+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:19.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:18.766+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:19.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:18.766+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:19.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:18.766+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:18.609+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:18.609+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:19.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:18.609+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:19.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:18.609+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:19.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:18.665+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:19.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:18.665+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:19.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:18.665+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:19.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:18.665+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:19.798+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:19.798+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:19.798+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:19.799+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:19.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:19.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:19.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:19.632+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:20.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:19.688+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:20.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:19.688+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:20.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:19.688+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:20.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:19.689+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:20.685+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:20.685+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:20.685+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:20.685+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:21.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:20.814+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:21.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:20.814+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:21.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:20.814+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:21.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:20.814+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:21.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:20.658+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:21.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:20.658+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:21.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:20.658+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:21.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:20.658+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:22.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:21.715+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:21.715+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:21.715+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:21.715+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:21.832+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:21.832+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:22.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:21.832+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:22.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:21.833+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:21.680+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:21.680+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:21.680+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:21.680+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:22.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:22.713+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:22.713+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:22.713+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:22.714+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:22.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:22.695+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:22.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:22.695+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:22.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:22.695+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:22.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:22.695+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:23.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:11:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:11:22.828+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 18475 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:11:23.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:22.868+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:23.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:22.868+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:23.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:22.868+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:23.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:22.868+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:23.918 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:23.756+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:23.919 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:23.756+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:23.919 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:23.756+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:23.919 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:23.757+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:23.919 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:23.711+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:23.919 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:23.711+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:23.919 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:23.711+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:23.920 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:23.711+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:24.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:23.918+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:23.918+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:23.918+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:23.919+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:25.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:24.870+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:25.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:24.870+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:25.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:24.870+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:25.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:24.871+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:24.737+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:24.737+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:25.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:24.737+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:25.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:24.737+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:25.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:24.750+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:25.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:24.750+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:25.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:24.750+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:25.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:24.751+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:25.898 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:25.799+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:25.899 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:25.799+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:25.899 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:25.799+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:25.899 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:25.800+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:25.899 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:25.750+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:25.899 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:25.750+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:25.899 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:25.750+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:25.900 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:25.750+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:26.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:25.898+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:26.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:25.898+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:26.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:25.898+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:26.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:25.899+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:27.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:26.881+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:27.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:26.881+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:27.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:26.881+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:27.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:26.882+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:26.762+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:27.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:26.762+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:27.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:26.762+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:27.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:26.763+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:27.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:26.715+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:27.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:26.715+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:27.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:26.715+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:27.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:26.715+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:28.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:11:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:11:27.830+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 18558 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:11:28.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:27.900+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:28.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:27.900+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:28.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:27.900+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:28.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:27.900+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:28.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:27.804+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:28.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:27.804+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:28.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:27.804+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:28.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:27.804+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:28.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:27.726+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:28.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:27.726+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:28.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:27.726+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:28.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:27.726+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:29.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:28.926+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:29.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:28.926+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:29.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:28.926+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:29.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:28.926+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:29.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:28.808+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:29.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:28.808+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:29.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:28.808+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:29.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:28.808+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:29.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:28.756+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:29.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:28.756+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:29.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:28.756+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:29.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:28.756+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:30.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:29.827+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:29.827+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:29.827+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:30.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:29.827+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:30.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:29.718+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:30.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:29.718+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:30.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:29.718+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:30.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:29.718+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:30.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:29.935+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:30.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:29.935+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:30.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:29.935+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:30.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:29.936+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:31.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:30.972+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:30.972+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:30.972+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:30.973+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:30.797+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:30.797+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:30.797+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:31.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:30.797+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:31.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:30.732+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:31.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:30.732+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:31.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:30.732+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:31.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:30.732+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:31.978 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:31.710+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:31.978 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:31.710+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:31.978 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:31.710+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:31.978 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:31.710+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:31.979 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:31.772+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:31.979 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:31.772+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:31.979 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:31.772+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:31.979 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:31.772+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:32.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:31.978+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:31.978+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:31.978+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:31.978+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:33.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:11:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:11:32.833+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 18640 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:11:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:32.985+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:32.985+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:32.985+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:32.986+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:33.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:32.765+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:33.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:32.765+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:33.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:32.765+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:33.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:32.766+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:33.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:32.750+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:33.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:32.750+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:33.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:32.750+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:33.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:32.751+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:34.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:33.731+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:33.731+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:33.731+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:33.731+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:34.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:33.791+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:34.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:33.791+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:34.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:33.791+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:34.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:33.791+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:34.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:34.005+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:34.005+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:34.005+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:34.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:34.006+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:35.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:34.823+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:34.823+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:34.823+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:35.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:34.823+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:34.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:34.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:34.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:35.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:34.711+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:35.038+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:35.038+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:35.038+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:35.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:35.039+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:36.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:35.833+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:35.833+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:35.833+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:36.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:35.834+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:35.674+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:35.674+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:35.674+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:36.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:35.674+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:36.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:36.080+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:36.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:36.080+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:36.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:36.080+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:36.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:36.081+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:37.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:36.835+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:37.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:36.835+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:37.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:36.835+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:37.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:36.835+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:36.639+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:36.639+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:37.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:36.639+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:37.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:36.639+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:37.076+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:37.076+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:37.076+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:37.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:37.076+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:37.959 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:11:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:11:37.836+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 18721 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:11:37.959 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:37.802+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:37.959 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:37.802+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:37.959 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:37.802+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:37.959 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:37.802+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:37.960 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:37.672+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:37.960 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:37.672+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:37.960 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:37.672+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:37.960 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:37.672+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:38.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:38.083+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:38.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:38.083+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:38.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:38.083+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:38.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:38.084+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:39.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:38.759+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:39.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:38.759+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:39.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:38.759+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:39.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:38.759+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:38.715+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:38.715+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:38.715+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:38.716+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:39.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:39.067+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:39.067+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:39.067+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:39.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:39.068+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:40.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:39.772+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:40.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:39.772+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:40.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:39.772+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:40.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:39.772+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:39.736+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:39.736+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:39.736+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:39.736+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:40.020+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:40.020+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:40.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:40.020+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:40.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:40.021+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:40.735+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:40.735+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:40.735+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:41.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:40.735+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:41.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:40.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:40.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:40.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:40.752+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:41.017+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:41.017+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:41.017+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:41.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:41.017+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:41.975 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:41.763+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:41.975 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:41.763+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:41.975 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:41.763+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:41.976 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:41.763+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:41.976 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:41.785+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:41.976 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:41.785+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:41.976 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:41.785+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:41.976 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:41.785+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:42.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:41.975+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:42.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:41.975+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:42.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:41.975+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:42.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:41.975+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:42.986 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:11:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:11:42.839+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 18803 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:11:42.987 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:42.732+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:42.987 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:42.732+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:42.987 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:42.732+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:42.987 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:42.732+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:42.988 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:42.818+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:42.988 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:42.818+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:42.988 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:42.818+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:42.988 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:42.818+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:43.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:42.986+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:43.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:42.986+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:43.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:42.986+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:43.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:42.987+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:43.987 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:43.725+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:43.987 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:43.725+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:43.988 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:43.725+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:43.988 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:43.726+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:43.988 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:43.834+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:43.988 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:43.834+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:43.988 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:43.834+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:43.988 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:43.834+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:44.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:44.019+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:44.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:44.019+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:44.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:44.019+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:44.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:44.020+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:44.742+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:44.742+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:44.742+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:44.742+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:45.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:44.856+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:45.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:44.856+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:45.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:44.856+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:45.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:44.856+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:45.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:45.019+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:45.019+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:45.019+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:45.019+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:46.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:45.985+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:46.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:45.985+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:46.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:45.985+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:46.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:45.985+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:46.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:45.791+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:46.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:45.791+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:46.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:45.791+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:46.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:45.792+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:46.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:45.855+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:46.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:45.855+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:46.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:45.855+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:46.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:45.855+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:47.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:47.025+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:47.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:47.025+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:47.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:47.025+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:47.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:47.026+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:47.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:46.831+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:47.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:46.831+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:47.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:46.831+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:47.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:46.831+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:47.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:46.826+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:47.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:46.826+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:47.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:46.826+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:47.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:46.827+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:48.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:11:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:11:47.841+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 18885 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:11:48.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:47.843+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:48.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:47.843+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:48.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:47.843+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:48.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:47.843+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:48.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:48.005+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:48.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:48.005+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:48.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:48.005+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:48.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:48.006+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:48.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:47.829+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:48.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:47.829+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:48.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:47.829+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:48.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:47.829+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:49.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:49.000+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:49.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:49.000+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:49.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:49.000+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:49.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:49.001+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:49.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:48.818+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:49.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:48.818+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:49.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:48.818+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:49.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:48.818+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:49.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:48.811+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:49.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:48.811+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:49.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:48.811+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:49.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:48.811+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:50.173 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:49.790+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:50.173 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:49.790+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:50.174 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:49.791+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:50.174 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:49.791+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:50.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:50.029+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:50.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:50.029+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:50.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:50.029+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:50.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:50.029+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:50.175 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:49.839+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:50.175 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:49.839+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:50.175 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:49.839+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:50.175 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:49.839+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:51.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:51.035+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:51.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:51.035+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:51.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:51.035+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:51.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:51.035+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:51.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:50.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:51.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:50.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:51.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:50.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:51.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:50.796+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:51.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:50.800+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:51.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:50.800+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:51.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:50.800+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:51.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:50.800+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:51.824+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:51.824+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:51.824+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:51.824+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:51.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:52.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:51.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:52.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:51.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:52.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:51.797+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:52.063+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:52.063+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:52.063+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:52.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:52.063+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:53.046 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:11:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:11:52.844+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 18967 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:11:53.046 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:52.842+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:53.046 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:52.842+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:53.046 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:52.842+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:53.047 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:52.842+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:53.047 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:52.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:53.047 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:52.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:53.047 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:52.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:53.047 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:52.761+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:53.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:53.046+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:53.046+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:53.046+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:53.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:53.046+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:54.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:53.764+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:53.764+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:53.764+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:53.764+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:53.792+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:54.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:53.792+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:54.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:53.792+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:54.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:53.793+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:54.057+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:54.057+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:54.057+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:54.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:54.057+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:55.010 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:54.828+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:55.011 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:54.828+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:55.011 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:54.828+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:55.011 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:54.828+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:55.011 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:54.760+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:55.011 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:54.760+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:55.012 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:54.760+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:55.012 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:54.760+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:55.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:55.011+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:55.011+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:55.011+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:55.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:55.011+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:56.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:55.986+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:55.986+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:55.986+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:56.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:55.986+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:56.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:55.786+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:56.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:55.786+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:56.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:55.786+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:56.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:55.786+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:56.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:55.815+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:56.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:55.815+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:56.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:55.815+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:56.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:55.815+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:57.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:56.942+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:56.942+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:56.942+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:56.943+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:57.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:56.816+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:57.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:56.816+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:57.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:56.816+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:57.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:56.817+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:57.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:56.815+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:57.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:56.815+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:57.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:56.815+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:24.244766+0000 (oldest deadline 2023-12-16T21:01:44.244766+0000) 2023-12-16T21:11:57.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:56.815+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:58.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:11:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:11:57.847+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 19048 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:11:58.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:57.807+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:58.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:57.979+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:58.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:57.979+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:58.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:57.979+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:58.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:57.979+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:58.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:57.840+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:58.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:57.840+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:58.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:57.840+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:58.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:57.840+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:59.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:58.854+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:59.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:58.867+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:59.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:58.867+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:59.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:58.867+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:11:59.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:58.867+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:11:59.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:58.960+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:58.960+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:58.960+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:11:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:58.960+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:00.173 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:11:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:11:59.849+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:00.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:59.993+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:12:00.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:59.993+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:12:00.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:59.993+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:12:00.174 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:11:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:11:59.994+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:00.174 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:59.865+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:00.174 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:59.865+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:00.175 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:59.865+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:00.175 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:11:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:11:59.865+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:01.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:00.877+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:01.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:00.906+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:01.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:00.906+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:01.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:00.906+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:01.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:00.906+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:01.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:01.029+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:12:01.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:01.029+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:12:01.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:01.029+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:28.396888+0000 (oldest deadline 2023-12-16T21:01:48.396888+0000) 2023-12-16T21:12:01.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:01.029+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:02.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:01.846+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:01.859+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:01.859+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:01.859+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:01.860+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:02.419 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:02.003+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:03.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:12:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:12:02.850+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 19130 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:12:03.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:03.035+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:03.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:02.881+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:03.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:02.834+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:03.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:02.834+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:03.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:02.834+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:03.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:02.834+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:03.868 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:03.835+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:04.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:04.054+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:04.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:03.869+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:04.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:03.869+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:04.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:03.869+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:04.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:03.869+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:05.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:04.848+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:05.264 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:05.059+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:05.264 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:04.884+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:05.264 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:04.884+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:05.264 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:04.884+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:05.264 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:04.884+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:06.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:05.864+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:06.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:06.013+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:06.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:05.904+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:06.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:05.904+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:06.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:05.904+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:06.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:05.905+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:07.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:06.998+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:07.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:06.911+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:07.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:06.906+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:07.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:06.906+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:07.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:06.906+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:01:32.994231+0000 (oldest deadline 2023-12-16T21:01:52.994231+0000) 2023-12-16T21:12:07.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:06.906+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:08.153 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:12:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:12:07.852+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 19211 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:12:08.153 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:08.030+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:08.154 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:07.939+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:08.154 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:07.868+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:09.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:09.073+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:09.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:08.942+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:09.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:08.852+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:10.175 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:09.841+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:10.175 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:10.058+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:10.175 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:09.989+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:11.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:11.000+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:11.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:11.033+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:11.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:10.834+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:12.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:11.957+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:11.840+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:12.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:12.050+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:13.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:12:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:12:12.855+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 19293 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:12:13.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:13.036+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:13.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:12.915+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:13.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:12.808+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:13.899 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:13.809+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:14.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:14.003+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:14.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:13.899+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:15.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:14.942+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:15.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:14.811+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:15.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:15.050+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:16.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:16.036+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:16.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:15.967+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:16.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:15.835+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:17.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:16.964+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:17.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:16.860+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:17.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:17.022+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:18.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:12:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:12:17.858+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 19374 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:12:18.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:18.046+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:18.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:17.985+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:18.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:17.838+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:19.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:19.094+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:19.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:19.011+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:19.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:18.847+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:20.175 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:19.985+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:20.176 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:20.063+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:20.176 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:19.886+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:20.176 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:19.886+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:20.176 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:19.886+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:20.176 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:19.886+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:21.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:21.046+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:21.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:20.955+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:21.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:20.915+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:21.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:20.915+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:21.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:20.915+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:21.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:20.915+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:22.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:21.953+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:21.874+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:21.875+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:21.875+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:21.875+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:22.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:22.015+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:23.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:12:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:12:22.861+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 19456 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:12:23.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:23.020+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:23.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:22.970+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:23.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:22.852+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:23.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:22.852+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:23.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:22.852+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:23.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:22.852+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:24.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:24.064+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:24.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:23.987+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:24.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:23.878+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:24.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:23.878+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:24.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:23.878+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:24.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:23.878+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:25.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:25.093+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:25.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:24.948+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:25.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:24.909+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:25.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:24.909+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:25.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:24.909+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:25.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:24.909+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:26.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:25.970+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:26.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:25.867+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:26.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:25.867+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:26.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:25.867+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:26.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:25.867+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:26.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:26.067+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:26.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:26.067+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:26.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:26.067+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:26.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:26.067+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:27.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:26.925+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:27.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:27.033+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:27.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:27.033+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:27.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:27.033+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:27.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:27.033+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:27.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:26.820+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:27.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:26.820+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:27.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:26.820+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:27.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:26.820+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:28.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:12:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:12:27.863+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 19537 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:12:28.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:27.963+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:28.062+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:28.062+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:28.062+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:28.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:28.062+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:27.840+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:27.840+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:27.840+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:27.840+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:29.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:29.027+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:29.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:29.027+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:29.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:29.027+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:29.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:29.027+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:29.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:28.991+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:29.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:28.991+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:29.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:28.991+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:29.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:28.992+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:29.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:28.793+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:29.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:28.793+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:29.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:28.793+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:29.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:28.793+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:30.176 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:29.962+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:30.177 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:29.963+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:30.177 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:29.963+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:30.177 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:29.963+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:30.177 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:29.991+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:30.177 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:29.991+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:30.177 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:29.991+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:30.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:29.991+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:30.178 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:29.842+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:30.178 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:29.842+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:30.178 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:29.842+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:30.178 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:29.842+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:31.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:30.987+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:31.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:30.987+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:31.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:30.987+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:31.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:30.987+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:31.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:31.011+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:31.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:31.011+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:31.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:31.011+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:31.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:31.012+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:31.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:30.863+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:31.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:30.863+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:31.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:30.863+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:31.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:30.863+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:31.965 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:31.841+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:31.965 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:31.841+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:31.966 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:31.841+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:31.966 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:31.841+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:32.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:32.021+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:32.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:32.021+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:32.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:32.021+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:32.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:32.021+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:31.965+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:31.965+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:31.965+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:31.965+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:33.139 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:12:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:12:32.866+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 19619 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:12:33.140 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:32.996+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:33.140 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:32.996+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:33.140 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:32.996+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:33.140 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:32.996+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:33.140 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:32.863+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:33.140 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:32.863+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:33.141 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:32.863+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:33.141 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:32.863+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:33.141 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:33.029+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:33.141 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:33.029+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:33.141 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:33.029+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:33.141 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:33.029+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:34.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:33.959+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:33.959+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:33.959+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:33.960+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:34.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:34.034+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:34.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:34.034+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:34.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:34.034+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:34.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:34.034+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:34.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:33.857+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:34.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:33.857+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:34.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:33.857+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:34.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:33.857+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:35.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:34.959+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:34.959+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:34.959+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:34.959+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:35.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:34.988+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:35.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:34.988+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:35.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:34.988+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:35.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:34.988+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:35.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:34.897+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:35.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:34.897+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:35.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:34.897+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:35.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:34.897+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:36.211 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:35.972+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:36.211 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:35.972+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:36.211 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:35.972+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:36.212 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:35.973+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:36.212 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:35.851+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:36.212 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:35.851+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:36.212 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:35.851+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:36.212 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:35.851+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:36.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:35.972+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:36.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:35.972+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:36.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:35.972+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:36.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:35.973+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:37.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:36.980+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:37.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:36.980+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:37.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:36.980+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:37.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:36.980+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:37.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:36.859+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:37.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:36.859+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:37.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:36.859+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:37.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:36.859+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:37.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:36.951+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:37.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:36.951+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:37.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:36.951+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:37.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:36.951+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:38.154 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:12:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:12:37.869+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 19700 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:12:38.154 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:37.886+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:38.155 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:37.886+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:38.155 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:37.886+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:38.155 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:37.886+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:38.155 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:37.937+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:38.155 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:37.937+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:38.156 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:37.937+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:38.156 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:37.938+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:38.156 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:37.932+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:38.156 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:37.932+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:38.156 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:37.932+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:38.156 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:37.932+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:39.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:38.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:39.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:38.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:39.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:38.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:39.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:38.988+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:39.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:38.901+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:39.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:38.901+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:39.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:38.901+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:39.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:38.902+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:39.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:38.919+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:39.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:38.919+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:39.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:38.919+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:39.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:38.919+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:40.177 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:39.892+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:40.177 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:39.892+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:40.177 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:39.892+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:40.177 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:39.892+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:40.177 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:39.989+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:40.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:39.989+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:40.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:39.989+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:40.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:39.989+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:40.178 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:39.943+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:40.178 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:39.943+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:40.178 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:39.943+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:40.179 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:39.943+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:41.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:41.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:41.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:41.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:41.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:41.039+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:41.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:41.040+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:41.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:40.940+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:41.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:40.940+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:41.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:40.940+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:41.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:40.941+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:41.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:40.890+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:41.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:40.890+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:41.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:40.890+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:41.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:40.890+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:41.996 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:41.915+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:41.996 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:41.915+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:41.996 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:41.915+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:41.997 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:41.915+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:41.997 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:41.880+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:41.997 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:41.880+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:41.997 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:41.880+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:41.997 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:41.881+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:42.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:41.995+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:42.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:41.995+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:42.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:41.996+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:42.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:41.996+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:43.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:12:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:12:42.873+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 19783 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:12:43.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:42.986+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:43.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:42.986+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:43.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:42.986+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:43.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:42.986+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:43.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:42.891+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:43.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:42.891+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:43.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:42.891+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:43.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:42.891+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:43.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:42.889+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:43.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:42.889+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:43.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:42.889+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:43.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:42.890+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:44.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:43.952+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:44.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:43.952+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:44.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:43.952+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:44.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:43.953+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:44.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:43.926+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:44.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:43.926+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:44.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:43.926+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:44.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:43.927+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:44.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:43.939+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:44.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:43.939+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:44.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:43.939+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:44.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:43.939+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:45.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:44.966+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:45.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:44.966+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:45.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:44.966+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:45.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:44.966+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:45.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:44.974+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:45.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:44.974+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:45.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:44.974+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:45.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:44.974+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:45.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:44.904+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:45.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:44.904+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:45.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:44.904+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:45.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:44.905+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:46.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:45.940+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:46.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:45.940+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:46.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:45.940+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:46.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:45.940+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:46.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:45.947+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:46.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:45.947+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:46.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:45.947+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:46.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:45.947+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:46.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:45.977+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:46.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:45.977+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:46.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:45.977+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:46.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:45.977+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:47.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:46.896+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:47.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:46.896+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:47.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:46.896+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:47.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:46.897+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:47.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:46.994+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:47.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:46.994+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:47.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:46.994+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:47.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:46.995+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:47.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:46.944+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:47.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:46.944+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:47.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:46.944+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:47.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:46.944+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:48.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:12:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:12:47.876+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 19866 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:12:48.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:47.905+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:48.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:47.905+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:48.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:47.905+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:48.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:47.905+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:48.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:47.984+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:48.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:47.984+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:48.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:47.984+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:48.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:47.984+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:48.255 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:47.949+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:48.255 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:47.949+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:48.255 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:47.949+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:48.256 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:47.949+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:49.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:49.018+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:49.018+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:49.018+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:49.019+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:49.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:48.910+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:49.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:48.910+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:49.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:48.910+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:49.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:48.910+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:49.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:48.936+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:49.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:48.936+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:49.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:48.936+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:49.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:48.937+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:50.177 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:50.030+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:50.177 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:50.030+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:50.177 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:50.030+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:50.177 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:50.031+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:50.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:49.953+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:50.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:49.953+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:50.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:49.953+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:50.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:49.954+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:50.178 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:49.912+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:50.179 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:49.912+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:50.179 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:49.912+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:50.179 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:49.912+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:51.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:51.061+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:51.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:51.061+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:51.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:51.061+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:51.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:51.061+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:51.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:50.951+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:51.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:50.951+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:51.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:50.951+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:51.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:50.951+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:51.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:50.936+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:51.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:50.936+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:51.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:50.936+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:51.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:50.936+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:52.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:51.925+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:52.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:51.925+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:52.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:51.925+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:52.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:51.926+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:52.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:52.073+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:52.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:52.073+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:52.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:52.073+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:52.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:52.074+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:52.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:51.943+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:52.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:51.943+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:52.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:51.943+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:52.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:51.943+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:53.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:12:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:12:52.879+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 19951 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:12:53.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:53.055+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:53.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:53.055+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:53.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:53.055+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:53.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:53.056+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:52.940+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:52.940+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:53.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:52.940+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:53.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:52.941+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:53.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:52.910+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:53.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:52.910+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:53.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:52.910+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:53.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:52.911+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:54.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:53.940+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:54.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:53.940+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:54.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:53.940+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:54.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:53.940+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:54.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:54.093+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:54.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:54.093+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:54.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:54.093+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:54.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:54.093+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:54.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:53.911+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:54.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:53.911+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:54.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:53.911+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:54.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:53.911+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:55.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:54.967+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:55.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:54.967+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:55.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:54.967+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:55.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:54.967+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:55.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:55.093+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:55.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:55.093+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:55.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:55.093+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:55.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:55.094+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:55.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:54.925+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:55.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:54.925+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:55.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:54.925+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:55.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:54.925+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:56.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:56.071+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:56.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:56.071+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:56.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:56.071+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:56.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:56.072+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:56.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:55.899+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:56.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:55.899+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:56.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:55.899+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:56.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:55.899+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:55.920+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:55.920+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:55.920+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:55.921+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:56.879+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:56.879+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:56.879+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:56.879+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:57.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:57.051+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:57.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:57.051+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:57.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:57.051+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:57.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:57.051+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:57.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:56.864+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:57.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:56.864+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:57.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:56.864+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:57.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:56.864+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:58.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:12:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:12:57.882+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 20034 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:12:58.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:57.841+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:58.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:57.841+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:58.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:57.841+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:58.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:57.841+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:58.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:57.874+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:58.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:57.874+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:58.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:57.874+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:58.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:57.874+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:58.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:58.016+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:58.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:58.017+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:58.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:58.017+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:58.255 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:58.017+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:59.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:58.841+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:59.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:58.841+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:59.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:58.841+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:12:59.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:58.841+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:58.905+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:58.905+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:58.905+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:12:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:58.906+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:12:59.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:58.995+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:59.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:58.995+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:59.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:58.995+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:12:59.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:58.995+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:00.177 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:59.882+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:00.177 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:59.882+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:00.178 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:59.882+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:00.178 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:12:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:12:59.882+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:00.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:59.940+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:00.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:59.940+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:00.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:59.940+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:00.179 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:12:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:12:59.940+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:00.179 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:59.977+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:00.179 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:59.977+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:00.179 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:59.977+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:00.179 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:12:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:12:59.977+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:01.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:00.936+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:01.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:00.936+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:01.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:00.936+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:01.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:00.937+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:01.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:00.961+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:01.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:00.961+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:01.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:00.961+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:01.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:00.961+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:01.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:00.909+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:01.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:00.909+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:01.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:00.909+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:01.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:00.909+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:01.964 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:01.863+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:01.964 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:01.863+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:01.964 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:01.863+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:01.965 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:01.863+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:01.965 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:01.944+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:01.965 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:01.944+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:01.965 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:01.944+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:01.965 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:01.944+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:02.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:01.964+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:02.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:01.964+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:02.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:01.964+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:02.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:01.964+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:03.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:13:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:13:02.885+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 20119 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:13:03.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:02.970+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:03.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:02.970+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:03.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:02.970+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:03.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:02.970+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:03.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:02.987+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:03.255 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:02.987+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:03.255 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:02.987+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:03.255 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:02.987+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:03.255 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:02.844+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:03.255 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:02.844+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:03.255 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:02.844+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:03.255 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:02.844+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:04.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:03.889+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:04.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:03.889+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:04.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:03.889+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:04.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:03.889+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:03.959+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:03.959+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:04.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:03.959+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:04.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:03.960+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:03.943+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:03.943+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:03.943+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:03.943+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:05.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:04.855+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:05.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:04.855+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:05.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:04.855+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:05.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:04.856+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:04.929+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:04.929+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:05.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:04.929+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:05.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:04.930+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:05.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:04.900+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:05.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:04.900+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:05.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:04.900+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:05.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:04.900+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:06.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:05.814+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:06.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:05.814+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:06.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:05.814+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:06.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:05.814+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:06.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:05.946+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:06.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:05.946+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:06.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:05.946+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:06.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:05.947+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:06.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:05.862+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:06.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:05.862+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:06.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:05.862+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:06.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:05.862+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:07.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:06.781+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:07.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:06.781+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:07.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:06.781+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:07.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:06.782+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:07.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:06.962+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:06.962+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:06.962+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:06.962+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:07.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:06.848+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:07.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:06.848+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:07.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:06.848+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:07.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:06.848+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:08.152 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:13:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:13:07.888+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 20203 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:13:08.153 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:07.786+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:08.153 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:07.786+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:08.153 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:07.786+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:08.153 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:07.786+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:08.153 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:07.939+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:08.154 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:07.939+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:08.154 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:07.939+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:08.154 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:07.939+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:08.154 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:07.811+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:08.154 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:07.811+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:08.154 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:07.812+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:08.155 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:07.812+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:09.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:08.846+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:09.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:08.846+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:09.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:08.846+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:09.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:08.846+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:09.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:08.810+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:09.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:08.810+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:09.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:08.810+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:09.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:08.810+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:08.902+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:08.902+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:08.902+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:09.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:08.902+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:10.177 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:09.878+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:10.178 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:09.878+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:10.178 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:09.878+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:10.178 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:09.879+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:10.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:09.917+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:10.178 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:09.917+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:10.179 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:09.917+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:10.179 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:09.918+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:10.179 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:09.851+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:10.179 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:09.851+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:10.179 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:09.851+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:10.179 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:09.851+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:11.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:10.832+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:11.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:10.832+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:11.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:10.832+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:11.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:10.833+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:11.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:10.903+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:11.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:10.903+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:11.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:10.903+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:11.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:10.904+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:11.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:10.820+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:11.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:10.820+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:11.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:10.820+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:11.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:10.821+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:12.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:11.869+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:11.869+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:11.869+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:12.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:11.869+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:11.910+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:11.910+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:11.910+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:11.911+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:12.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:11.811+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:12.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:11.811+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:12.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:11.811+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:12.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:11.811+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:13.075 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:13:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:13:12.891+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 20288 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:13:13.076 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:12.830+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:13.076 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:12.830+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:13.076 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:12.830+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:13.076 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:12.830+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:13.076 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:12.774+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:13.077 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:12.774+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:13.077 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:12.774+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:13.077 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:12.774+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:13.077 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:12.901+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:13.077 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:12.901+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:13.077 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:12.901+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:13.078 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:12.901+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:13.872 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:13.726+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:13.872 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:13.726+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:13.873 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:13.726+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:13.873 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:13.726+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:14.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:13.937+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:14.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:13.937+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:14.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:13.937+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:14.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:13.937+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:14.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:13.872+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:14.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:13.872+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:14.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:13.872+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:14.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:13.872+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:14.957 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:14.842+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:14.958 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:14.842+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:14.958 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:14.842+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:14.958 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:14.842+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:14.959 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:14.678+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:14.959 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:14.678+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:14.959 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:14.678+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:14.959 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:14.678+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:15.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:14.957+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:14.957+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:14.957+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:14.957+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:15.978 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:15.727+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:15.978 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:15.728+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:15.978 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:15.728+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:15.979 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:15.728+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:15.979 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:15.873+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:15.979 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:15.873+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:15.979 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:15.873+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:15.979 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:15.874+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:16.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:15.978+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:16.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:15.978+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:16.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:15.978+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:15.978+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:17.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:16.702+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:16.702+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:16.702+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:16.702+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:16.932+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:17.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:16.932+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:17.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:16.932+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:17.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:16.932+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:17.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:16.902+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:17.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:16.902+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:17.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:16.902+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:17.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:16.903+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:18.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:13:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:13:17.894+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 20371 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:13:18.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:17.751+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:18.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:17.751+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:18.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:17.751+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:18.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:17.751+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:18.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:17.927+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:18.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:17.927+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:18.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:17.927+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:18.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:17.928+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:18.006 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:17.922+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:18.006 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:17.922+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:18.006 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:17.922+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:18.007 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:17.922+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:19.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:18.879+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:18.879+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:18.879+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:18.879+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:18.745+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:19.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:18.745+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:19.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:18.745+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:19.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:18.745+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:19.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:18.940+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:19.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:18.940+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:19.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:18.940+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:19.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:18.940+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:20.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:19.729+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:20.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:19.729+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:20.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:19.729+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:20.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:19.729+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:19.848+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:20.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:19.848+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:19.848+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:19.848+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:20.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:19.947+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:20.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:19.947+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:20.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:19.947+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:20.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:19.947+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:21.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:20.680+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:20.680+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:20.680+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:21.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:20.680+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:20.887+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:21.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:20.887+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:21.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:20.887+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:21.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:20.887+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:21.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:20.899+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:21.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:20.899+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:21.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:20.899+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:21.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:20.900+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:22.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:21.682+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:21.682+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:21.682+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:21.682+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:21.885+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:21.885+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:22.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:21.885+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:22.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:21.886+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:21.916+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:21.916+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:21.916+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:21.916+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:22.949 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:13:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:13:22.897+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 20457 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:13:22.950 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:22.684+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:22.950 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:22.684+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:22.951 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:22.684+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:22.951 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:22.684+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:22.951 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:22.885+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:22.951 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:22.885+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:22.952 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:22.885+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:22.952 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:22.885+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:23.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:22.950+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:23.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:22.950+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:23.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:22.950+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:23.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:22.950+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:23.871 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:23.720+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:23.871 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:23.720+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:23.871 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:23.720+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:23.871 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:23.720+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:23.871+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:23.871+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:23.871+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:23.871+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:24.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:23.919+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:24.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:23.919+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:24.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:23.919+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:24.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:23.919+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:25.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:24.881+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:24.882+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:24.882+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:25.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:24.882+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:25.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:24.873+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:25.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:24.873+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:25.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:24.873+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:25.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:24.873+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:25.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:24.674+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:25.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:24.674+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:25.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:24.674+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:25.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:24.675+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:25.924 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:25.665+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:25.924 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:25.665+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:25.924 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:25.665+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:25.924 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:25.665+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:25.924 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:25.887+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:25.924 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:25.887+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:25.925 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:25.887+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:25.925 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:25.887+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:26.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:25.923+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:26.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:25.923+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:26.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:25.923+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:26.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:25.924+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:26.919 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:26.657+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:26.920 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:26.657+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:26.920 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:26.657+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:26.920 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:26.657+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:26.920 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:26.857+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:26.920 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:26.857+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:26.921 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:26.857+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:26.921 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:26.858+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:27.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:26.919+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:27.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:26.919+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:27.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:26.919+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:27.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:26.920+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:27.901 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:27.622+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:27.901 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:27.622+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:27.902 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:27.622+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:27.902 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:27.622+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:28.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:13:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:13:27.900+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 20540 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:13:28.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:27.939+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:28.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:27.939+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:28.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:27.939+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:28.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:27.940+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:28.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:27.907+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:28.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:27.907+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:28.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:27.907+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:28.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:27.908+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:28.909 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:28.877+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:28.910 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:28.877+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:28.910 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:28.877+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:28.910 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:28.877+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:28.910 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:28.634+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:28.910 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:28.634+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:28.910 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:28.634+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:28.910 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:28.634+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:29.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:28.909+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:29.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:28.909+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:29.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:28.909+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:29.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:28.909+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:29.949 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:29.876+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:29.949 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:29.876+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:29.949 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:29.876+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:29.949 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:29.877+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:29.950 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:29.667+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:29.950 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:29.667+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:29.950 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:29.667+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:29.950 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:29.667+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:30.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:29.949+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:30.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:29.949+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:30.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:29.949+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:30.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:29.949+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:30.887+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:30.887+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:30.887+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:30.887+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:31.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:30.928+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:31.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:30.928+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:31.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:30.928+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:31.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:30.928+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:31.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:30.690+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:31.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:30.690+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:31.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:30.690+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:31.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:30.690+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:31.887 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:31.734+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:31.887 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:31.734+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:31.887 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:31.734+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:31.887 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:31.735+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:32.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:31.887+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:31.887+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:31.887+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:31.887+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:32.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:31.900+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:32.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:31.900+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:32.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:31.900+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:32.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:31.900+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:33.002 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:13:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:13:32.903+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 20625 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:13:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:32.869+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:32.869+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:32.869+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:32.869+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:33.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:32.911+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:33.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:32.911+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:33.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:32.911+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:33.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:32.911+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:33.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:32.723+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:33.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:32.723+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:33.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:32.723+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:33.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:32.723+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:33.867 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:33.755+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:33.867 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:33.755+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:33.867 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:33.755+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:33.868 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:33.755+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:34.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:33.867+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:33.867+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:33.867+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:33.867+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:34.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:33.904+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:34.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:33.904+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:34.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:33.904+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:34.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:33.904+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:34.870 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:34.718+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:34.870 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:34.718+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:34.870 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:34.718+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:34.871 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:34.718+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:35.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:34.870+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:34.870+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:34.870+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:35.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:34.870+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:35.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:34.938+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:35.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:34.938+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:35.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:34.938+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:35.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:34.938+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:35.960 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:35.831+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:35.961 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:35.831+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:35.961 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:35.831+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:35.961 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:35.831+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:35.961 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:35.681+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:35.961 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:35.681+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:35.961 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:35.681+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:35.962 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:35.681+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:36.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:35.961+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:36.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:35.961+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:36.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:35.961+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:36.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:35.961+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:37.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:36.932+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:36.932+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:36.932+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:36.932+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:37.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:36.799+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:37.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:36.799+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:37.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:36.799+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:37.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:36.799+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:37.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:36.705+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:37.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:36.705+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:37.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:36.705+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:37.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:36.705+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:38.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:13:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:13:37.907+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 20708 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:13:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:37.884+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:37.884+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:37.884+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:38.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:37.884+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:37.829+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:37.829+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:37.829+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:37.829+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:38.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:37.710+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:38.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:37.710+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:38.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:37.710+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:38.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:37.710+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:39.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:38.806+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:38.806+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:38.806+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:38.806+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:38.861+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:38.861+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:38.861+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:39.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:38.862+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:39.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:38.710+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:39.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:38.710+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:39.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:38.710+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:39.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:38.710+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:40.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:39.828+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:40.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:39.828+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:40.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:39.828+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:40.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:39.829+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:39.861+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:39.861+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:39.861+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:39.862+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:40.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:39.745+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:40.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:39.745+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:40.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:39.745+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:40.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:39.745+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:41.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:40.872+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:41.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:40.872+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:41.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:40.872+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:41.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:40.872+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:40.852+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:40.852+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:40.852+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:40.852+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:41.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:40.722+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:41.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:40.722+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:41.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:40.722+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:41.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:40.722+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:41.859+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:41.859+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:41.859+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:42.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:41.859+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:42.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:41.811+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:42.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:41.811+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:42.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:41.811+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:42.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:41.812+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:42.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:41.725+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:42.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:41.725+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:42.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:41.725+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:42.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:41.725+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:43.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:13:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:13:42.910+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 20794 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:13:43.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:42.853+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:43.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:42.853+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:43.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:42.853+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:43.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:42.854+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:43.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:42.853+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:43.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:42.853+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:43.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:42.853+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:43.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:42.853+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:43.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:42.725+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:43.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:42.725+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:43.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:42.725+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:43.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:42.725+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:43.875 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:43.814+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:43.875 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:43.814+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:43.876 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:43.814+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:43.876 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:43.814+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:43.876 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:43.734+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:43.876 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:43.734+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:43.876 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:43.734+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:43.876 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:43.734+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:44.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:43.875+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:44.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:43.875+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:44.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:43.875+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:44.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:43.875+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:45.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:44.848+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:45.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:44.848+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:45.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:44.848+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:45.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:44.849+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:44.847+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:44.847+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:45.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:44.847+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:45.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:44.848+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:45.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:44.739+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:45.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:44.739+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:45.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:44.740+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:45.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:44.740+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:46.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:45.810+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:45.810+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:45.810+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:45.810+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:46.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:45.703+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:46.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:45.703+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:46.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:45.703+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:46.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:45.703+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:46.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:45.808+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:46.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:45.808+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:46.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:45.808+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:46.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:45.808+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:47.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:46.769+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:46.769+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:46.769+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:46.769+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:47.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:46.706+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:47.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:46.706+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:47.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:46.706+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:47.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:46.707+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:47.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:46.774+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:47.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:46.774+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:47.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:46.774+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:47.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:46.774+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:48.001 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:13:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:13:47.912+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 20877 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:13:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:47.802+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:47.802+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:47.802+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:48.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:47.802+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:47.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:47.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:47.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:47.802+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:48.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:47.744+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:48.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:47.744+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:48.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:47.744+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:48.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:47.744+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:49.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:48.755+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:49.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:48.755+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:49.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:48.755+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:49.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:48.755+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:48.753+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:48.753+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:49.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:48.753+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:49.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:48.754+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:49.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:48.751+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:49.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:48.751+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:49.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:48.751+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:49.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:48.751+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:50.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:49.709+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:50.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:49.709+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:50.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:49.709+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:50.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:49.709+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:50.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:49.745+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:50.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:49.745+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:50.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:49.745+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:50.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:49.745+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:50.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:49.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:50.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:49.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:50.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:49.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:50.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:49.752+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:51.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:50.742+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:51.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:50.742+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:51.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:50.742+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:51.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:50.742+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:50.711+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:51.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:50.711+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:51.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:50.711+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:51.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:50.711+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:51.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:50.733+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:51.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:50.733+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:51.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:50.733+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:51.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:50.733+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:52.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:51.694+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:51.694+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:51.694+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:51.694+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:52.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:51.734+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:52.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:51.734+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:52.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:51.734+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:52.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:51.734+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:51.744+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:51.744+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:51.744+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:52.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:51.745+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:52.938 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:13:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:13:52.915+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 20963 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:13:52.938 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:52.687+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:52.938 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:52.687+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:52.938 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:52.687+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:52.938 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:52.688+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:52.939 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:52.685+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:52.939 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:52.685+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:52.939 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:52.685+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:52.939 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:52.685+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:52.939 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:52.775+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:52.940 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:52.775+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:52.940 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:52.775+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:52.940 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:52.775+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:53.723+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:53.723+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:53.723+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:53.723+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:53.706+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:54.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:53.706+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:54.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:53.706+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:54.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:53.706+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:54.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:53.812+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:54.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:53.812+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:54.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:53.812+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:54.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:53.813+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:55.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:54.696+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:54.696+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:54.696+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:55.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:54.696+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:55.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:54.844+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:55.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:54.844+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:55.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:54.844+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:55.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:54.844+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:55.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:54.662+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:55.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:54.662+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:55.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:54.662+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:55.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:54.662+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:56.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:55.659+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:56.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:55.659+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:55.659+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:56.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:55.660+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:56.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:55.893+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:56.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:55.893+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:56.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:55.893+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:55.893+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:56.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:55.671+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:56.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:55.671+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:56.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:55.671+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:56.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:55.671+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:56.919 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:56.657+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:56.919 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:56.657+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:56.920 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:56.657+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:56.920 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:56.657+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:56.920 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:56.686+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:56.920 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:56.686+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:56.920 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:56.686+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:56.920 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:56.686+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:57.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:56.919+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:56.919+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:56.919+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:57.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:56.919+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:57.934 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:13:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:13:57.918+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 21043 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:13:57.935 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:57.722+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:57.935 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:57.722+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:57.935 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:57.722+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:57.935 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:57.722+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:57.935 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:57.673+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:57.935 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:57.673+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:57.936 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:57.673+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:57.936 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:57.673+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:57.936 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:57.876+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:57.936 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:57.876+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:57.936 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:57.876+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:57.936 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:57.876+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:58.846+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:58.846+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:58.846+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:13:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:58.846+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:58.736+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:58.736+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:59.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:58.736+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:13:59.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:58.736+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:13:59.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:58.636+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:59.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:58.636+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:59.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:58.636+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:13:59.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:58.636+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:00.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:59.800+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:00.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:59.800+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:00.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:59.800+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:00.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:13:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:13:59.800+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:00.006 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:59.722+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:00.007 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:59.722+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:00.007 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:59.722+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:00.007 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:13:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:13:59.722+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:00.007 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:59.627+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:00.007 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:59.627+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:00.007 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:59.627+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:00.007 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:13:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:13:59.627+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:01.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:00.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:01.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:00.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:01.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:00.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:01.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:00.802+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:00.693+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:00.693+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:01.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:00.693+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:01.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:00.693+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:01.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:00.667+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:01.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:00.667+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:01.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:00.667+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:01.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:00.667+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:02.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:01.790+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:01.790+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:01.790+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:01.790+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:01.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:01.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:02.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:01.727+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:02.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:01.727+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:02.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:01.632+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:02.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:01.632+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:02.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:01.632+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:02.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:01.632+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:02.922 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:02.731+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:02.923 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:02.731+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:02.923 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:02.731+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:02.923 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:02.731+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:02.923 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:02.660+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:02.924 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:02.660+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:02.924 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:02.660+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:02.924 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:02.660+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:02.924 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:02.826+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:02.924 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:02.826+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:02.924 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:02.826+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:02.924 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:02.826+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:03.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:14:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:14:02.922+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 21121 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:14:04.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:03.797+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:03.797+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:03.797+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:03.798+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:03.702+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:03.702+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:04.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:03.702+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:04.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:03.702+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:04.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:03.653+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:04.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:03.653+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:04.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:03.653+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:04.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:03.654+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:05.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:04.827+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:05.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:04.827+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:05.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:04.827+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:05.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:04.827+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:04.657+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:04.657+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:04.657+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:04.658+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:05.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:04.628+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:05.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:04.628+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:05.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:04.628+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:05.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:04.628+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:06.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:05.673+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:06.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:05.673+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:06.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:05.673+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:06.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:05.674+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:06.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:05.836+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:06.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:05.836+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:06.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:05.836+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:06.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:05.837+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:06.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:05.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:06.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:05.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:06.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:05.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:06.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:05.615+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:06.885 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:06.566+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:06.885 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:06.566+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:06.885 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:06.566+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:06.885 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:06.567+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:06.885 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:06.711+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:06.886 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:06.711+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:06.886 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:06.711+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:06.886 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:06.711+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:07.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:06.884+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:06.884+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:06.884+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:06.885+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:07.880 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:07.555+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:07.880 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:07.555+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:07.880 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:07.555+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:07.881 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:07.555+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:07.881 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:07.662+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:07.881 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:07.662+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:07.881 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:07.662+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:07.881 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:07.663+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:08.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:14:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:14:07.925+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 21194 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:14:08.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:07.880+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:08.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:07.880+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:08.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:07.880+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:08.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:07.880+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:08.857 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:08.546+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:08.858 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:08.546+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:08.858 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:08.546+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:08.858 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:08.547+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:08.858 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:08.630+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:08.858 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:08.630+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:08.858 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:08.630+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:08.858 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:08.630+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:09.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:08.857+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:09.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:08.857+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:09.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:08.857+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:09.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:08.857+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:09.846 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:09.518+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:09.847 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:09.518+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:09.847 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:09.518+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:09.847 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:09.518+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:09.847 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:09.615+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:09.847 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:09.615+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:09.848 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:09.615+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:09.848 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:09.615+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:10.181 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:09.846+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:10.181 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:09.846+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:10.181 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:09.846+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:10.182 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:09.846+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:10.837 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:10.506+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:10.837 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:10.506+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:10.837 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:10.506+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:10.838 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:10.506+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:10.838 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:10.656+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:10.838 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:10.656+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:10.838 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:10.656+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:10.838 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:10.656+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:11.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:10.837+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:11.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:10.837+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:11.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:10.837+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:11.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:10.837+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:11.717 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:11.641+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:11.717 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:11.641+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:11.717 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:11.641+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:11.718 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:11.641+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:11.718 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:11.553+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:11.718 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:11.553+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:11.718 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:11.553+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:11.718 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:11.553+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:12.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:11.808+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:12.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:11.808+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:12.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:11.808+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:12.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:11.808+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:12.834 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:12.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:12.834 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:12.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:12.834 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:12.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:12.834 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:12.512+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:12.835 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:12.642+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:12.835 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:12.642+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:12.835 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:12.642+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:12.835 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:12.642+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:13.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:14:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:14:12.928+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 21271 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:14:13.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:12.833+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:13.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:12.833+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:13.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:12.833+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:13.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:12.834+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:13.843 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:13.557+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:13.843 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:13.557+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:13.843 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:13.557+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:13.843 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:13.557+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:13.843 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:13.646+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:13.844 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:13.646+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:13.844 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:13.646+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:13.844 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:13.646+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:14.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:13.842+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:14.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:13.842+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:14.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:13.842+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:14.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:13.843+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:14.800 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:14.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:14.800 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:14.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:14.800 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:14.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:14.801 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:14.606+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:14.801 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:14.610+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:14.801 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:14.610+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:14.801 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:14.610+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:14.802 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:14.610+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:15.251 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:14.799+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:14.799+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:14.799+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:14.800+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:16.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:15.752+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:16.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:15.752+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:16.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:15.752+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:16.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:15.752+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:15.568+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:15.568+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:16.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:15.568+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:16.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:15.569+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:16.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:15.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:16.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:15.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:16.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:15.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:16.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:15.642+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:17.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:16.707+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:16.707+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:16.707+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:16.707+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:17.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:16.615+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:17.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:16.615+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:17.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:16.615+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:17.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:16.615+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:17.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:16.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:17.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:16.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:17.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:16.606+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:17.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:16.607+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:17.932 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:17.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:17.932 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:17.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:17.932 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:17.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:17.932 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:17.631+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:17.933 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:17.578+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:17.933 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:17.578+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:17.933 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:17.578+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:17.933 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:17.578+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:17.934 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:17.693+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:17.934 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:17.693+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:17.934 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:17.693+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:17.934 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:17.694+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:18.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:14:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:14:17.932+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 21345 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:14:19.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:18.591+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:18.591+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:18.591+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:18.591+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:18.539+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:18.539+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:19.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:18.539+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:19.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:18.539+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:19.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:18.678+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:19.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:18.678+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:19.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:18.678+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:19.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:18.678+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:19.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:19.636+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:19.636+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:19.636+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:19.637+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:19.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:19.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:19.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:19.631+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:19.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:19.632+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:19.499+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:19.499+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:19.499+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:19.499+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:20.651+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:20.651+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:20.651+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:20.651+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:20.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:20.496+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:20.496+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:20.496+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:20.496+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:20.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:20.610+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:20.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:20.610+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:20.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:20.610+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:20.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:20.611+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:21.717 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:21.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:21.718 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:21.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:21.718 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:21.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:21.718 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:21.665+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:21.718 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:21.453+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:21.718 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:21.453+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:21.719 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:21.453+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:21.719 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:21.453+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:21.719 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:21.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:21.719 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:21.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:21.719 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:21.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:21.719 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:21.564+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:22.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:22.407+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:22.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:22.407+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:22.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:22.407+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:22.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:22.407+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:22.638+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:22.638+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:22.638+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:22.639+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:22.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:22.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:22.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:22.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:22.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:22.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:22.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:22.577+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:23.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:14:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:14:22.935+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 21425 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:14:23.731 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:23.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:23.731 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:23.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:23.731 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:23.590+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:23.731 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:23.591+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:23.732 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:23.570+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:23.732 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:23.570+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:23.732 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:23.570+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:23.732 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:23.571+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:23.732 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:23.382+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:23.732 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:23.382+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:23.733 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:23.382+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:23.733 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:23.382+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:24.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:24.597+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:24.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:24.597+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:24.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:24.597+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:24.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:24.598+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:24.566+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:24.566+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:24.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:24.566+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:24.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:24.566+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:24.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:24.404+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:24.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:24.404+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:24.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:24.404+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:24.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:24.404+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:25.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:25.438+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:25.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:25.438+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:25.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:25.438+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:25.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:25.438+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:25.530+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:25.530+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:25.530+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:25.531+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:25.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:25.629+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:25.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:25.629+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:25.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:25.629+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:25.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:25.629+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:26.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:26.412+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:26.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:26.412+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:26.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:26.412+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:26.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:26.413+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:26.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:26.616+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:26.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:26.616+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:26.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:26.616+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:26.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:26.617+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:26.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:26.557+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:26.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:26.557+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:26.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:26.557+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:26.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:26.557+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:27.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:27.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:27.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:27.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:27.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:27.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:27.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:27.587+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:27.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:27.409+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:27.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:27.409+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:27.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:27.409+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:27.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:27.409+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:27.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:27.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:27.558+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:27.558+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:28.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:14:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:14:27.938+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 21505 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:14:28.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:28.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:28.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:28.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:28.529+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:28.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:28.424+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:28.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:28.424+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:28.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:28.424+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:28.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:28.424+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:28.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:28.609+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:28.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:28.609+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:28.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:28.609+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:28.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:28.609+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:29.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:29.573+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:29.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:29.573+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:29.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:29.573+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:29.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:29.573+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:29.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:29.437+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:29.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:29.437+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:29.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:29.437+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:29.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:29.437+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:29.481+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:29.481+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:29.481+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:29.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:29.482+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:30.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:30.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:30.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:30.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:30.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:30.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:30.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:30.389+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:30.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:30.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:30.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:30.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:30.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:30.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:30.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:30.454+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:30.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:30.531+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:30.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:30.531+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:30.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:30.531+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:30.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:30.532+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:31.720 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:31.436+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:31.720 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:31.436+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:31.720 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:31.436+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:31.720 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:31.436+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:31.720 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:31.342+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:31.721 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:31.342+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:31.721 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:31.342+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:31.721 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:31.342+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:31.721 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:31.562+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:31.721 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:31.562+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:31.721 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:31.562+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:31.721 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:31.562+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:32.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:32.471+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:32.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:32.471+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:32.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:32.471+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:32.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:32.471+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:32.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:32.608+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:32.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:32.608+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:32.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:32.608+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:32.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:32.608+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:32.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:32.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:32.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:32.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:32.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:32.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:32.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:32.369+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:33.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:14:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:14:32.941+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 21591 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:14:33.613 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:33.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:33.613 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:33.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:33.613 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:33.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:33.613 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:33.564+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:33.613 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:33.449+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:33.614 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:33.449+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:33.614 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:33.449+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:33.614 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:33.450+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:33.614 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:33.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:33.614 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:33.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:33.614 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:33.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:33.614 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:33.356+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:34.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:34.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:34.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:34.563+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:34.498+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:34.498+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:34.498+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:34.498+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:34.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:34.322+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:34.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:34.322+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:34.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:34.322+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:34.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:34.322+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:35.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:35.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:35.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:35.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:35.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:35.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:35.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:35.541+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:35.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:35.537+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:35.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:35.537+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:35.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:35.537+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:35.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:35.537+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:35.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:35.341+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:35.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:35.341+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:35.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:35.341+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:35.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:35.341+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:36.584 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:36.323+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:36.584 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:36.323+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:36.584 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:36.323+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:36.584 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:36.323+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:36.585 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:36.548+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:36.585 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:36.548+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:36.585 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:36.548+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:36.585 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:36.548+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:37.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:36.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:36.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:36.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:37.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:36.584+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:37.585 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:37.303+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:37.586 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:37.303+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:37.586 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:37.303+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:37.586 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:37.303+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:37.945 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:37.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:37.945 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:37.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:37.945 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:37.618+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:37.946 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:37.618+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:37.946 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:37.586+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:37.946 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:37.586+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:37.946 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:37.586+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:37.946 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:37.586+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:38.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:14:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:14:37.944+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 21673 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:14:38.538 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:38.270+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:38.539 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:38.270+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:38.539 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:38.270+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:38.539 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:38.271+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:39.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:38.538+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:38.538+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:38.538+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:38.539+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:38.647+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:38.647+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:38.647+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:38.647+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:39.651 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:39.308+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:39.651 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:39.308+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:39.651 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:39.308+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:39.651 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:39.308+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:39.652 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:39.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:39.652 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:39.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:39.652 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:39.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:39.652 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:39.520+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:40.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:39.651+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:39.651+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:39.651+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:39.651+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:40.495 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:40.320+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:40.495 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:40.320+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:40.495 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:40.320+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:40.495 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:40.320+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:40.495+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:40.495+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:40.495+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:40.495+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:40.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:40.669+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:40.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:40.669+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:40.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:40.669+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:40.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:40.669+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:41.707 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:41.513+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:41.707 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:41.513+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:41.707 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:41.513+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:41.707 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:41.514+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:41.708 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:41.315+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:41.708 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:41.315+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:41.708 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:41.315+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:41.708 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:41.315+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:42.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:41.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:41.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:41.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:41.707+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:42.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:42.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:42.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:42.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:42.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:42.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:42.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:42.307+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:42.757 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:42.502+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:42.757 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:42.502+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:42.758 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:42.502+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:42.758 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:42.503+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:43.075 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:14:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:14:42.948+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 21758 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:14:43.075 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:42.758+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:43.075 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:42.758+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:43.075 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:42.758+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:43.075 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:42.758+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:43.459 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:43.269+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:43.460 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:43.269+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:43.460 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:43.269+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:43.460 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:43.269+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:43.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:43.459+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:43.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:43.459+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:43.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:43.459+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:43.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:43.460+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:44.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:43.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:44.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:43.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:44.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:43.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:44.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:43.761+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:44.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:44.443+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:44.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:44.443+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:44.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:44.443+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:44.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:44.444+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:44.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:44.309+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:44.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:44.309+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:44.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:44.309+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:44.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:44.309+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:45.235 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:44.763+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:45.235 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:44.763+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:45.236 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:44.763+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:45.236 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:44.764+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:45.436+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:45.436+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:45.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:45.436+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:45.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:45.436+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:45.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:45.318+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:45.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:45.318+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:45.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:45.318+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:45.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:45.319+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:46.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:45.744+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:45.744+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:45.744+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:45.744+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:46.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:46.418+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:46.418+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:46.418+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:46.418+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:46.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:46.273+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:46.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:46.273+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:46.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:46.273+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:46.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:46.273+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:47.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:46.788+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:47.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:46.788+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:47.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:46.788+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:47.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:46.789+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:47.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:47.441+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:47.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:47.441+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:47.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:47.441+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:47.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:47.441+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:47.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:47.306+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:47.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:47.306+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:47.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:47.306+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:47.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:47.306+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:48.009 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:14:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:14:47.951+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 21840 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:14:48.009 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:47.754+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:48.009 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:47.754+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:48.009 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:47.754+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:48.010 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:47.755+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:48.406+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:48.406+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:48.406+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:48.406+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:48.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:48.338+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:48.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:48.338+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:48.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:48.338+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:48.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:48.338+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:49.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:48.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:48.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:48.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:48.762+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:49.749 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:49.384+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:49.749 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:49.384+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:49.749 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:49.384+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:49.749 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:49.385+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:49.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:49.378+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:49.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:49.378+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:49.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:49.378+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:49.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:49.378+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:50.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:49.749+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:49.749+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:49.749+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:49.749+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:50.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:50.379+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:50.379+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:50.379+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:50.380+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:50.394+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:50.394+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:50.394+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:50.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:50.394+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:51.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:50.761+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:51.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:50.762+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:51.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:50.762+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:51.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:50.762+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:51.719 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:51.344+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:51.719 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:51.344+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:51.719 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:51.344+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:51.719 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:51.345+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:51.720 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:51.363+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:51.720 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:51.363+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:51.720 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:51.363+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:51.720 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:51.363+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:51.726+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:51.726+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:51.726+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:52.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:51.726+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:52.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:52.338+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:52.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:52.338+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:52.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:52.338+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:52.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:52.339+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:52.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:52.381+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:52.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:52.381+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:52.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:52.381+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:52.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:52.382+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:53.081 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:14:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:14:52.954+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 21926 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:14:53.081 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:52.754+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:53.082 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:52.754+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:53.082 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:52.754+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:53.082 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:52.754+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:53.357 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:53.317+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:53.357 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:53.317+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:53.357 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:53.317+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:53.357 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:53.317+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:53.614 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:53.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:53.614 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:53.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:53.615 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:53.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:53.615 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:53.357+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:53.797+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:53.797+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:53.797+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:53.797+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:54.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:54.360+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:54.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:54.360+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:54.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:54.360+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:54.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:54.360+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:54.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:54.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:54.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:54.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:54.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:54.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:54.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:54.398+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:55.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:54.757+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:55.251 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:54.758+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:55.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:54.758+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:55.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:54.758+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:55.721 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:55.386+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:55.721 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:55.386+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:55.721 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:55.386+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:55.722 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:55.386+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:55.722 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:55.333+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:55.722 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:55.333+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:55.722 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:55.333+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:55.722 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:55.334+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:56.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:55.719+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:56.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:55.720+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:56.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:55.720+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:56.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:55.720+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:56.348+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:56.348+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:56.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:56.348+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:56.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:56.349+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:56.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:56.362+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:56.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:56.362+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:56.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:56.362+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:56.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:56.362+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:57.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:56.757+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:57.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:56.757+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:57.255 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:56.757+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:57.255 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:56.757+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:57.713 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:57.402+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:57.713 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:57.402+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:57.714 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:57.402+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:57.714 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:57.403+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:57.714 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:57.318+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:57.714 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:57.318+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:57.714 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:57.318+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:57.714 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:57.318+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:57.969 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:14:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:14:57.958+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 22009 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:14:57.969 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:57.713+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:57.969 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:57.713+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:57.969 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:57.713+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:57.970 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:57.714+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:58.685 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:58.400+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:58.686 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:58.400+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:58.686 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:58.400+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:58.686 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:58.400+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:58.686 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:58.283+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:58.686 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:58.283+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:58.686 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:58.283+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:58.687 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:58.283+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:59.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:58.685+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:58.685+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:58.685+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:14:59.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:58.686+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:59.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:59.411+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:59.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:59.411+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:59.658 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:59.411+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:14:59.658 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:14:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:14:59.411+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:14:59.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:59.283+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:59.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:59.283+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:59.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:59.283+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:14:59.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:14:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:14:59.283+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:00.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:59.657+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:59.657+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:59.657+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:00.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:14:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:14:59.658+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:00.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:00.417+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:00.417+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:00.417+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:00.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:00.418+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:00.258+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:00.258+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:00.258+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:00.258+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:01.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:00.687+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:00.687+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:00.687+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:01.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:00.687+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:01.668 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:01.399+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:01.669 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:01.399+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:01.669 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:01.399+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:01.669 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:01.399+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:01.669 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:01.296+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:01.669 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:01.297+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:01.669 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:01.297+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:01.670 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:01.297+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:02.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:01.668+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:01.668+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:01.668+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:02.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:01.669+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:02.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:02.399+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:02.399+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:02.399+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:02.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:02.400+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:02.250+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:02.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:02.250+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:02.250+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:02.251+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:02.961 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:02.657+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:02.962 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:02.657+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:02.962 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:02.657+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:02.962 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:02.657+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:03.215 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:15:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:15:02.961+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 22095 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:15:03.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:03.354+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:03.354+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:03.354+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:03.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:03.355+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:03.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:03.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:03.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:03.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:03.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:03.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:03.216+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:04.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:03.690+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:03.690+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:03.690+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:03.690+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:04.373+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:04.373+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:04.373+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:04.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:04.373+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:04.214+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:04.214+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:04.214+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:04.215+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:05.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:04.663+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:04.663+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:04.663+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:04.663+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:05.422 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:05.167+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:05.422 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:05.167+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:05.422 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:05.167+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:05.422 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:05.167+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:05.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:05.635+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:05.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:05.635+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:05.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:05.635+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:05.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:05.635+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:05.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:05.422+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:05.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:05.422+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:05.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:05.422+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:05.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:05.422+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:06.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:06.157+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:06.157+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:06.157+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:06.157+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:06.403+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:06.403+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:06.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:06.403+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:06.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:06.403+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:06.608+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:06.608+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:06.608+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:06.608+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:07.425 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:07.122+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:07.425 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:07.122+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:07.425 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:07.122+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:07.425 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:07.122+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:07.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:07.424+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:07.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:07.424+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:07.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:07.424+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:07.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:07.424+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:07.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:07.632+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:07.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:07.632+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:07.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:07.632+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:07.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:07.632+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:08.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:15:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:15:07.963+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 22178 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:15:08.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:08.118+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:08.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:08.118+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:08.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:08.118+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:08.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:08.118+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:08.646 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:08.412+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:08.647 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:08.412+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:08.647 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:08.412+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:08.647 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:08.412+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:08.646+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:08.646+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:08.646+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:09.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:08.646+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:09.457 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:09.154+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:09.457 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:09.154+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:09.457 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:09.154+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:09.457 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:09.154+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:09.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:09.456+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:09.456+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:09.456+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:09.456+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:09.670+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:09.670+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:09.670+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:09.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:09.670+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:10.451 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:10.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:10.451 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:10.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:10.451 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:10.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:10.452 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:10.129+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:10.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:10.449+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:10.449+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:10.449+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:10.450+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:10.697+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:10.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:10.697+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:10.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:10.697+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:10.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:10.697+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:11.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:11.137+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:11.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:11.137+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:11.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:11.137+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:11.465 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:11.137+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:11.720 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:11.464+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:11.720 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:11.464+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:11.720 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:11.464+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:11.720 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:11.464+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:11.721 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:11.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:11.721 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:11.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:11.721 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:11.710+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:11.721 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:11.710+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:12.459 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:12.172+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:12.459 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:12.172+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:12.459 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:12.172+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:12.459 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:12.172+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:12.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:12.458+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:12.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:12.458+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:12.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:12.458+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:12.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:12.458+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:12.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:12.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:12.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:12.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:12.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:12.707+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:12.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:12.707+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:13.187 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:15:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:15:12.967+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 22264 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:15:13.445 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:13.187+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:13.445 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:13.187+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:13.446 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:13.187+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:13.446 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:13.187+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:13.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:13.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:13.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:13.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:13.662+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:13.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:13.445+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:13.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:13.445+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:13.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:13.445+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:13.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:13.445+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:14.480 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:14.207+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:14.480 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:14.207+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:14.480 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:14.207+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:14.480 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:14.207+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:14.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:14.479+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:14.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:14.479+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:14.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:14.479+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:14.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:14.479+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:14.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:14.617+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:14.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:14.617+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:14.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:14.617+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:14.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:14.617+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:15.501 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:15.472+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:15.472+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:15.472+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:15.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:15.472+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:15.244+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:15.244+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:15.244+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:15.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:15.244+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:15.955 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:15.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:15.955 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:15.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:15.956 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:15.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:15.956 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:15.606+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:16.629 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:16.267+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:16.630 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:16.267+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:16.630 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:16.267+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:16.630 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:16.267+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:16.631 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:16.466+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:16.631 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:16.466+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:16.631 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:16.466+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:16.631 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:16.466+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:17.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:16.626+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:16.626+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:16.626+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:17.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:16.627+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:17.612 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:17.467+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:17.612 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:17.467+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:17.612 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:17.467+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:17.612 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:17.467+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:17.612 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:17.275+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:17.612 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:17.275+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:17.613 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:17.275+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:17.613 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:17.275+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:17.972 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:17.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:17.972 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:17.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:17.972 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:17.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:17.972 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:17.611+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:18.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:15:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:15:17.970+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 22346 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:15:18.597 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:18.498+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:18.597 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:18.498+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:18.597 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:18.498+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:18.597 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:18.498+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:18.597 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:18.287+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:18.598 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:18.287+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:18.598 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:18.287+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:18.598 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:18.287+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:19.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:18.596+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:19.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:18.596+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:18.596+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:19.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:18.596+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:19.527 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:19.255+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:19.527 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:19.255+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:19.527 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:19.255+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:19.527 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:19.255+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:19.525+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:19.525+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:19.525+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:20.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:19.526+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:19.548+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:19.548+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:19.548+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:19.548+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:20.496 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:20.287+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:20.496 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:20.287+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:20.497 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:20.287+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:20.497 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:20.287+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:20.496+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:20.496+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:20.496+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:20.496+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:20.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:20.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:20.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:20.530+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:21.561 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:21.501+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:21.561 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:21.501+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:21.561 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:21.501+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:21.561 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:21.501+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:21.562 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:21.254+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:21.562 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:21.254+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:21.562 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:21.254+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:21.562 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:21.254+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:22.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:21.560+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:21.560+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:21.560+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:21.560+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:22.481 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:22.235+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:22.482 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:22.235+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:22.482 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:22.235+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:22.482 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:22.235+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:22.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:22.481+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:22.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:22.481+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:22.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:22.481+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:22.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:22.481+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:22.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:22.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:22.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:22.572+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:23.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:15:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:15:22.974+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 22431 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:15:23.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:23.198+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:23.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:23.198+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:23.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:23.198+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:23.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:23.198+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:23.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:23.485+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:23.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:23.485+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:23.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:23.485+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:23.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:23.486+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:23.547+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:23.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:23.548+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:23.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:23.548+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:23.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:23.548+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:24.482 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:24.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:24.483 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:24.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:24.483 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:24.180+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:24.483 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:24.180+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:24.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:24.544+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:24.544+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:24.544+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:24.544+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:24.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:24.482+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:24.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:24.482+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:24.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:24.482+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:24.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:24.483+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:25.457 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:25.194+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:25.458 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:25.194+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:25.458 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:25.194+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:25.458 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:25.194+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:25.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:25.457+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:25.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:25.457+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:25.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:25.457+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:25.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:25.458+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:25.532+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:25.532+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:25.532+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:25.533+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:26.422+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:26.422+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:26.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:26.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:26.423+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:26.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:26.182+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:26.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:26.182+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:26.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:26.182+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:26.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:26.182+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:27.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:26.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:26.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:26.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:27.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:26.571+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:27.439 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:27.147+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:27.439 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:27.147+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:27.439 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:27.147+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:27.439 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:27.148+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:27.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:27.438+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:27.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:27.438+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:27.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:27.438+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:27.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:27.439+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:27.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:27.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:27.572+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:27.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:27.572+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:28.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:15:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:15:27.978+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 22513 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:15:28.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:28.107+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:28.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:28.107+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:28.107+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:28.108+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:28.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:28.569+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:28.569+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:28.569+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:28.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:28.570+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:28.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:28.461+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:28.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:28.461+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:28.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:28.461+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:28.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:28.461+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:29.493 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:29.073+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:29.493 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:29.073+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:29.493 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:29.073+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:29.493 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:29.073+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:29.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:29.522+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:29.522+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:29.522+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:29.523+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:29.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:29.493+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:29.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:29.493+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:29.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:29.493+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:29.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:29.493+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:30.093+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:30.093+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:30.093+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:30.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:30.093+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:31.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:30.538+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:30.538+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:30.538+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:30.538+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:30.527+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:31.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:30.527+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:31.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:30.527+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:31.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:30.527+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:31.315 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:31.061+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:31.315 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:31.061+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:31.315 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:31.061+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:31.316 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:31.061+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:31.566 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:31.538+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:31.566 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:31.538+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:31.566 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:31.538+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:31.566 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:31.538+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:32.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:31.565+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:31.565+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:31.565+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:32.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:31.566+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:32.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:32.055+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:32.055+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:32.055+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:32.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:32.055+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:32.982 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:32.542+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:32.982 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:32.542+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:32.982 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:32.542+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:32.982 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:32.542+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:32.983 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:32.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:32.983 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:32.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:32.983 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:32.605+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:32.983 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:32.605+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:33.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:15:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:15:32.982+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 22599 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:15:33.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:33.043+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:33.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:33.043+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:33.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:33.043+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:33.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:33.043+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:34.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:33.568+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:33.568+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:33.568+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:34.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:33.569+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:33.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:33.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:34.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:33.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:34.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:33.576+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:34.387 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:34.091+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:34.387 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:34.091+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:34.388 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:34.091+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:34.388 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:34.092+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:34.564+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:34.564+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:34.564+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:34.565+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:34.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:34.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:34.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:34.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:34.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:34.577+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:35.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:35.127+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:35.127+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:35.127+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:35.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:35.127+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:36.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:35.596+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:35.596+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:35.596+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:36.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:35.597+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:35.569+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:35.569+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:35.569+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:35.570+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:36.154+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:36.154+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:36.154+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:36.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:36.154+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:37.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:36.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:37.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:36.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:37.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:36.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:37.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:36.564+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:37.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:36.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:37.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:36.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:37.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:36.594+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:37.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:36.595+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:37.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:37.125+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:37.125+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:37.125+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:37.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:37.126+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:37.986 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:37.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:37.986 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:37.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:37.986 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:37.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:37.986 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:37.633+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:37.986 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:37.543+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:37.987 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:37.543+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:37.987 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:37.543+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:37.987 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:37.543+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:38.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:15:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:15:37.985+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 22681 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:15:38.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:38.159+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:38.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:38.159+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:38.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:38.159+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:38.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:38.159+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:38.551+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:38.551+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:38.551+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:38.551+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:39.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:38.639+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:39.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:38.639+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:39.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:38.639+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:39.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:38.640+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:39.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:39.114+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:39.114+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:39.114+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:39.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:39.114+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:40.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:39.518+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:40.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:39.518+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:40.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:39.518+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:40.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:39.518+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:40.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:39.677+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:39.677+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:39.677+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:39.678+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:40.070+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:40.070+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:40.070+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:40.070+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:41.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:40.549+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:41.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:40.549+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:41.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:40.549+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:41.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:40.549+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:40.702+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:40.702+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:40.702+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:40.702+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:41.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:41.072+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:41.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:41.072+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:41.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:41.072+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:41.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:41.072+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:42.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:41.556+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:41.556+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:41.556+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:41.557+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:41.699+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:42.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:41.699+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:42.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:41.699+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:42.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:41.699+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:42.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:42.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:42.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:42.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:42.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:42.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:42.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:42.072+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:42.990 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:42.734+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:42.990 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:42.734+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:42.990 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:42.734+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:42.990 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:42.734+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:42.991 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:42.527+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:42.991 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:42.527+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:42.991 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:42.527+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:42.991 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:42.527+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:43.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:15:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:15:42.989+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 22767 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:15:43.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:43.122+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:43.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:43.122+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:43.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:43.122+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:43.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:43.122+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:43.775+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:43.775+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:43.775+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:44.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:43.776+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:44.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:43.535+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:44.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:43.535+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:44.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:43.535+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:44.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:43.535+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:44.167+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:44.167+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:44.167+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:44.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:44.167+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:45.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:44.782+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:44.782+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:44.782+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:45.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:44.782+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:45.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:44.545+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:45.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:44.545+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:44.545+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:44.546+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:45.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:45.130+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:45.130+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:45.130+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:45.130+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:45.759+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:45.759+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:45.759+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:46.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:45.760+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:46.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:45.534+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:46.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:45.534+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:46.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:45.534+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:46.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:45.535+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:46.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:46.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:46.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:46.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:46.131+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:46.721+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:46.721+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:46.721+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:47.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:46.721+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:47.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:46.561+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:47.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:46.561+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:47.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:46.561+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:47.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:46.561+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:47.147+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:47.147+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:47.147+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:47.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:47.147+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:47.993 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:47.676+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:47.993 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:47.676+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:47.994 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:47.676+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:47.994 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:47.676+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:47.994 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:47.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:47.994 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:47.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:47.994 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:47.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:47.995 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:47.576+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:48.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:15:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:15:47.992+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 22850 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:15:48.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:48.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:48.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:48.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:48.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:48.129+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:48.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:48.129+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:49.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:48.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:49.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:48.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:49.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:48.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:49.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:48.540+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:49.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:48.697+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:49.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:48.697+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:49.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:48.697+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:49.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:48.697+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:49.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:49.103+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:49.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:49.103+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:49.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:49.103+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:49.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:49.103+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:50.001 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:49.538+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:50.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:49.538+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:50.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:49.538+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:50.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:49.539+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:49.729+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:50.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:49.729+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:50.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:49.729+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:50.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:49.729+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:50.060+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:50.060+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:50.060+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:50.060+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:51.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:50.543+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:51.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:50.543+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:51.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:50.543+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:51.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:50.543+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:51.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:50.684+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:51.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:50.684+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:51.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:50.684+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:51.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:50.684+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:51.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:51.103+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:51.103+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:51.103+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:51.103+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:51.505+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:51.505+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:51.505+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:51.505+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:52.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:51.715+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:52.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:51.715+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:52.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:51.715+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:52.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:51.716+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:52.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:52.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:52.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:52.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:52.071+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:52.819 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:52.723+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:52.820 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:52.723+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:52.820 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:52.723+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:52.820 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:52.723+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:52.821 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:52.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:52.821 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:52.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:52.821 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:52.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:52.822 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:52.516+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:53.121 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:15:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:15:52.996+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 22936 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:15:53.122 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:53.047+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:53.122 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:53.047+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:53.122 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:53.047+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:53.123 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:53.048+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:53.739 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:53.487+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:53.739 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:53.487+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:53.739 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:53.487+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:53.739 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:53.488+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:54.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:53.739+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:53.739+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:53.739+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:54.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:53.739+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:54.453 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:54.021+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:54.453 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:54.021+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:54.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:54.021+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:54.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:54.021+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:54.737 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:54.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:54.737 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:54.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:54.737 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:54.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:54.738 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:54.453+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:55.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:54.985+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:55.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:54.985+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:55.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:54.985+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:55.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:54.985+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:55.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:54.737+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:55.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:54.737+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:55.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:54.737+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:55.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:54.737+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:55.722 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:55.447+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:55.722 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:55.447+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:55.722 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:55.447+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:55.722 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:55.447+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:55.941+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:55.941+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:55.941+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:56.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:55.941+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:56.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:55.722+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:56.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:55.722+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:56.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:55.722+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:56.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:55.722+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:56.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:56.430+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:56.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:56.430+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:56.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:56.430+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:56.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:56.431+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:56.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:56.675+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:56.675+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:56.675+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:56.675+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:57.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:56.912+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:57.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:56.912+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:57.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:56.912+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:57.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:56.912+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:57.656 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:57.395+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:57.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:57.395+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:57.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:57.395+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:57.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:57.395+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:57.961 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:57.656+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:57.962 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:57.656+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:57.962 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:57.656+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:57.962 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:57.656+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:58.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:15:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:15:58.000+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 23019 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:15:58.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:57.961+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:58.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:57.961+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:58.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:57.961+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:58.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:57.961+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:58.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:58.389+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:58.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:58.389+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:58.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:58.389+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:58.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:58.390+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:58.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:58.624+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:58.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:58.624+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:58.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:58.624+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:58.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:58.624+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:59.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:58.914+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:59.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:58.914+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:59.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:58.914+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:15:59.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:58.914+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:59.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:59.419+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:59.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:59.419+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:59.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:59.419+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:15:59.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:15:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:15:59.419+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:15:59.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:59.600+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:59.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:59.600+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:59.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:59.600+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:15:59.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:15:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:15:59.600+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:00.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:59.898+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:00.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:59.898+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:00.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:59.898+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:00.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:15:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:15:59.899+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:00.596+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:00.596+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:00.596+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:00.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:00.596+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:00.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:00.429+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:00.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:00.429+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:00.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:00.429+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:00.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:00.430+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:01.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:00.871+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:01.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:00.871+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:01.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:00.871+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:01.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:00.871+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:01.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:01.582+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:01.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:01.582+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:01.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:01.582+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:01.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:01.582+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:01.724 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:01.440+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:01.725 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:01.440+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:01.725 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:01.440+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:01.725 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:01.441+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:02.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:01.837+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:02.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:01.837+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:02.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:01.837+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:02.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:01.837+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:02.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:02.405+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:02.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:02.405+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:02.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:02.405+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:02.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:02.406+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:02.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:02.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:02.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:02.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:02.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:02.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:02.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:02.612+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:03.133 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:16:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:16:03.003+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 23105 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:16:03.134 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:02.820+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:03.134 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:02.820+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:03.134 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:02.820+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:03.134 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:02.820+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:03.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:03.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:03.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:03.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:03.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:03.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:03.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:03.642+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:03.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:03.447+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:03.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:03.447+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:03.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:03.447+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:03.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:03.448+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:04.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:03.868+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:04.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:03.868+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:04.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:03.868+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:04.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:03.868+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:04.687 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:04.431+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:04.687 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:04.431+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:04.687 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:04.431+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:04.688 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:04.431+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:05.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:04.687+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:04.687+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:04.687+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:05.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:04.687+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:05.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:04.866+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:05.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:04.866+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:05.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:04.866+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:05.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:04.866+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:05.694 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:05.414+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:05.694 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:05.414+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:05.694 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:05.414+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:05.694 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:05.414+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:05.694+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:05.694+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:05.694+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:05.694+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:06.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:05.900+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:06.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:05.900+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:06.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:05.900+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:06.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:05.900+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:06.732 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:06.391+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:06.732 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:06.391+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:06.732 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:06.391+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:06.732 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:06.392+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:07.001 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:06.732+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:06.732+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:06.732+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:07.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:06.732+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:07.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:06.857+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:07.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:06.857+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:07.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:06.857+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:07.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:06.857+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:07.678 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:07.439+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:07.679 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:07.439+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:07.679 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:07.439+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:07.679 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:07.440+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:07.742+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:07.742+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:07.742+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:08.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:07.742+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:08.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:07.902+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:08.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:07.902+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:08.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:07.902+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:08.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:07.903+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:08.468 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:16:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:16:08.005+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 23189 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:16:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:08.466+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:08.466+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:08.466+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:08.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:08.466+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:09.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:08.771+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:09.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:08.771+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:09.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:08.771+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:09.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:08.772+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:09.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:08.858+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:09.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:08.858+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:09.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:08.858+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:09.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:08.858+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:09.420+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:09.420+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:09.420+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:09.420+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:10.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:09.767+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:10.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:09.767+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:10.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:09.767+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:10.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:09.767+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:10.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:09.896+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:10.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:09.896+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:10.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:09.896+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:10.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:09.896+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:10.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:10.385+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:10.385+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:10.385+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:10.385+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:11.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:10.802+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:11.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:10.802+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:11.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:10.802+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:11.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:10.802+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:11.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:10.946+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:11.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:10.946+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:11.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:10.946+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:11.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:10.946+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:11.724 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:11.398+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:11.724 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:11.398+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:11.725 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:11.398+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:11.725 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:11.398+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:11.916+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:11.916+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:11.916+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:12.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:11.917+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:12.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:11.803+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:12.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:11.803+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:12.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:11.803+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:12.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:11.803+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:12.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:12.368+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:12.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:12.368+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:12.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:12.368+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:12.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:12.368+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:13.032 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:16:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:16:13.009+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 23275 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:16:13.033 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:12.938+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:13.033 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:12.938+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:13.033 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:12.938+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:13.033 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:12.938+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:13.034 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:12.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:13.034 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:12.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:13.034 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:12.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:13.034 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:12.777+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:13.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:13.394+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:13.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:13.394+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:13.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:13.394+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:13.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:13.394+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:14.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:13.769+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:14.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:13.769+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:14.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:13.769+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:14.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:13.769+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:14.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:13.953+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:14.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:13.953+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:14.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:13.953+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:14.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:13.953+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:14.746 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:14.391+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:14.746 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:14.391+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:14.747 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:14.391+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:14.747 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:14.392+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:15.001 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:14.923+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:15.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:14.923+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:15.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:14.923+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:15.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:14.923+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:14.746+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:15.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:14.746+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:15.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:14.746+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:15.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:14.746+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:15.743 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:15.442+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:15.743 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:15.442+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:15.743 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:15.442+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:15.743 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:15.442+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:15.743+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:15.743+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:15.743+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:16.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:15.743+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:16.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:15.878+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:16.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:15.878+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:16.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:15.878+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:16.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:15.879+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:16.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:16.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:16.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:16.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:16.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:16.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:16.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:16.454+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:17.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:16.887+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:17.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:16.887+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:17.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:16.887+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:17.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:16.887+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:17.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:16.787+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:17.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:16.787+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:17.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:16.787+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:17.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:16.787+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:17.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:17.450+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:17.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:17.450+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:17.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:17.450+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:17.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:17.450+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:18.142 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:16:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:16:18.013+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 23358 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:16:18.143 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:17.879+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:18.143 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:17.879+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:18.143 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:17.879+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:18.144 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:17.879+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:18.144 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:17.816+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:18.144 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:17.816+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:18.144 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:17.816+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:18.144 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:17.816+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:18.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:18.411+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:18.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:18.411+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:18.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:18.411+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:18.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:18.412+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:19.213 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:18.862+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:19.213 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:18.862+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:19.213 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:18.862+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:19.213 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:18.863+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:19.214 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:18.858+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:19.214 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:18.858+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:19.214 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:18.858+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:19.214 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:18.859+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:19.391+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:19.391+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:19.391+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:19.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:19.391+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:20.251 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:19.869+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:20.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:19.869+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:20.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:19.869+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:20.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:19.869+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:20.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:19.901+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:20.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:19.901+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:20.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:19.901+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:20.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:19.902+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:20.342+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:20.342+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:20.343+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:20.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:20.343+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:21.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:20.850+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:21.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:20.850+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:21.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:20.850+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:21.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:20.850+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:21.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:20.940+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:21.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:20.940+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:21.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:20.940+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:21.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:20.940+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:21.724 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:21.299+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:21.725 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:21.299+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:21.725 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:21.299+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:21.725 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:21.299+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:21.860+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:21.860+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:21.860+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:21.860+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:21.914+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:21.914+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:21.914+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:21.914+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:22.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:22.312+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:22.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:22.312+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:22.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:22.312+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:22.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:22.312+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:23.142 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:16:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:16:23.016+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 23444 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:16:23.142 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:22.821+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:23.143 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:22.821+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:23.143 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:22.821+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:23.143 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:22.821+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:23.143 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:22.927+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:23.143 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:22.927+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:23.143 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:22.927+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:23.143 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:22.927+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:23.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:23.358+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:23.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:23.358+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:23.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:23.358+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:23.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:23.358+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:24.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:23.907+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:24.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:23.907+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:24.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:23.907+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:24.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:23.908+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:24.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:23.815+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:24.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:23.815+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:24.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:23.815+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:24.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:23.815+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:24.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:24.393+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:24.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:24.393+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:24.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:24.393+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:24.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:24.393+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:25.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:24.906+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:25.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:24.906+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:25.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:24.906+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:25.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:24.906+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:25.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:24.860+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:25.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:24.860+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:25.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:24.860+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:25.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:24.860+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:25.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:25.359+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:25.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:25.359+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:25.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:25.359+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:25.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:25.359+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:26.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:25.911+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:26.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:25.911+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:26.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:25.911+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:26.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:25.911+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:26.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:25.906+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:26.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:25.906+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:26.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:25.906+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:26.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:25.906+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:26.611 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:26.343+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:26.612 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:26.343+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:26.612 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:26.343+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:26.612 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:26.343+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:27.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:26.928+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:27.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:26.928+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:27.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:26.928+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:27.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:26.928+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:27.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:26.929+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:27.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:26.929+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:27.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:26.929+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:27.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:26.929+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:27.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:27.354+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:27.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:27.354+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:27.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:27.354+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:27.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:27.354+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:28.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:16:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:16:28.020+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 23527 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:16:28.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:27.938+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:27.938+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:27.938+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:28.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:27.939+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:28.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:27.930+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:28.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:27.930+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:28.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:27.930+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:28.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:27.930+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:28.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:28.376+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:28.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:28.376+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:28.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:28.376+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:28.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:28.376+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:29.209 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:28.966+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:29.209 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:28.966+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:29.210 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:28.966+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:29.210 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:28.966+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:29.210 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:28.934+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:29.210 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:28.934+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:29.210 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:28.934+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:29.210 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:28.934+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:29.329+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:29.329+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:29.329+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:29.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:29.330+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:30.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:29.981+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:30.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:29.981+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:30.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:29.981+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:30.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:29.981+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:30.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:30.002+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:30.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:30.002+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:30.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:30.002+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:30.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:30.002+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:30.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:30.323+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:30.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:30.323+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:30.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:30.323+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:30.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:30.324+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:31.306 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:31.014+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:31.306 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:31.014+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:31.307 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:31.014+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:31.307 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:31.014+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:31.307 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:31.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:31.307 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:31.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:31.307 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:31.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:31.307 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:31.049+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:31.726 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:31.305+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:31.726 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:31.305+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:31.726 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:31.305+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:31.726 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:31.305+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:32.280 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:32.051+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:32.280 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:32.051+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:32.280 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:32.051+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:32.280 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:32.051+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:32.281 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:32.024+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:32.281 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:32.024+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:32.281 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:32.024+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:32.281 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:32.024+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:32.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:32.279+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:32.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:32.279+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:32.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:32.279+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:32.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:32.279+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:33.007 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:32.983+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:33.007 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:32.983+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:33.007 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:32.983+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:33.007 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:32.983+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:33.323 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:16:33.022+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 23613 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:16:33.323 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:33.006+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:33.323 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:33.006+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:33.323 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:33.006+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:33.323 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:33.006+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:33.622 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:33.322+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:33.622 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:33.322+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:33.622 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:33.322+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:33.622 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:33.322+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:34.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:33.994+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:34.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:33.994+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:34.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:33.994+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:34.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:33.995+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:34.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:33.994+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:34.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:33.994+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:34.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:33.994+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:34.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:33.994+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:34.288+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:34.288+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:34.288+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:34.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:34.288+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:35.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:34.977+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:35.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:34.977+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:35.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:34.977+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:35.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:34.977+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:35.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:35.041+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:35.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:35.041+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:35.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:35.041+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:35.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:35.041+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:35.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:35.331+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:35.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:35.331+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:35.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:35.331+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:35.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:35.331+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:36.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:35.987+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:36.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:35.987+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:36.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:35.987+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:36.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:35.987+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:36.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:36.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:36.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:36.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:36.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:36.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:36.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:36.067+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:36.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:36.319+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:36.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:36.319+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:36.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:36.319+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:36.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:36.319+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:37.347 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:37.036+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:37.347 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:37.036+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:37.347 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:37.036+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:37.348 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:37.036+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:37.348 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:37.092+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:37.348 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:37.092+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:37.348 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:37.092+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:37.348 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:37.092+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:37.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:37.347+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:37.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:37.347+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:37.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:37.347+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:37.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:37.347+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:38.388 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:16:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:16:38.026+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 23696 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:16:38.389 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:38.037+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:38.389 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:38.037+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:38.389 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:38.037+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:38.389 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:38.037+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:38.390 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:38.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:38.390 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:38.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:38.390 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:38.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:38.390 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:38.133+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:38.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:38.387+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:38.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:38.387+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:38.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:38.387+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:38.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:38.388+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:39.389 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:39.023+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:39.390 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:39.023+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:39.390 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:39.023+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:39.390 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:39.023+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:39.390 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:39.111+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:39.390 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:39.111+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:39.390 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:39.111+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:39.391 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:39.111+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:39.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:39.388+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:39.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:39.388+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:39.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:39.388+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:39.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:39.388+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:40.418 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:40.038+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:40.418 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:40.038+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:40.419 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:40.038+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:40.419 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:40.038+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:40.419 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:40.102+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:40.419 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:40.102+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:40.419 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:40.102+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:40.419 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:40.102+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:40.417+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:40.417+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:40.417+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:40.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:40.417+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:41.442 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:41.005+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:41.443 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:41.005+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:41.443 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:41.005+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:41.443 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:41.005+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:41.443 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:41.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:41.443 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:41.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:41.443 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:41.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:41.444 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:41.113+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:41.726 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:41.442+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:41.726 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:41.442+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:41.726 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:41.442+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:41.726 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:41.442+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:42.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:41.991+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:42.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:41.991+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:42.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:41.991+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:42.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:41.991+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:42.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:42.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:42.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:42.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:42.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:42.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:42.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:42.088+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:42.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:42.412+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:42.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:42.412+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:42.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:42.412+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:42.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:42.412+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:43.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:16:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:16:43.030+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 23782 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:16:43.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:43.064+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:43.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:43.064+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:43.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:43.064+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:43.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:43.064+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:43.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:42.962+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:43.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:42.962+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:43.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:42.962+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:43.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:42.962+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:43.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:43.373+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:43.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:43.373+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:43.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:43.373+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:43.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:43.374+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:44.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:43.985+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:44.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:43.985+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:44.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:43.985+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:44.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:43.985+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:44.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:44.101+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:44.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:44.101+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:44.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:44.101+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:44.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:44.101+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:44.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:44.352+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:44.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:44.352+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:44.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:44.352+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:44.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:44.352+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:45.340 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:45.018+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:45.340 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:45.018+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:45.340 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:45.018+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:45.340 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:45.019+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:45.340 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:45.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:45.341 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:45.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:45.341 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:45.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:45.341 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:45.131+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:45.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:45.339+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:45.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:45.339+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:45.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:45.339+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:45.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:45.339+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:46.389 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:46.003+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:46.389 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:46.003+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:46.389 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:46.003+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:46.389 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:46.004+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:46.390 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:46.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:46.390 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:46.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:46.390 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:46.141+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:46.390 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:46.141+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:46.388+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:46.388+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:46.388+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:46.389+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:47.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:46.981+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:47.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:46.981+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:47.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:46.981+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:47.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:46.982+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:47.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:47.092+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:47.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:47.092+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:47.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:47.092+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:47.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:47.093+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:47.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:47.438+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:47.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:47.438+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:47.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:47.438+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:47.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:47.438+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:48.467 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:16:48.034+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 23865 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:16:48.468 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:48.018+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:48.468 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:48.018+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:48.468 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:48.018+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:48.468 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:48.019+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:48.468 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:48.078+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:48.469 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:48.078+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:48.469 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:48.078+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:48.469 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:48.078+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:48.466+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:48.466+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:48.466+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:48.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:48.467+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:48.986+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:48.986+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:48.986+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:49.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:48.986+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:49.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:49.086+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:49.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:49.086+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:49.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:49.086+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:49.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:49.087+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:49.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:49.486+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:49.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:49.486+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:49.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:49.486+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:49.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:49.486+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:50.453 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:50.009+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:50.453 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:50.009+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:50.453 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:50.009+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:50.454 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:50.009+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:50.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:50.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:50.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:50.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:50.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:50.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:50.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:50.071+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:50.452+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:50.452+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:50.452+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:50.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:50.452+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:51.459 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:51.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:51.459 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:51.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:51.459 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:51.049+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:51.459 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:51.050+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:51.460 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:51.010+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:51.460 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:51.010+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:51.460 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:51.010+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:51.460 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:51.010+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:51.727 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:51.458+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:51.727 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:51.458+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:51.727 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:51.458+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:51.727 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:51.459+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:52.459 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:52.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:52.460 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:52.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:52.460 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:52.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:52.460 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:52.071+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:52.460 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:52.025+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:52.461 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:52.025+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:52.461 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:52.025+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:52.461 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:52.025+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:52.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:52.459+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:52.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:52.459+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:52.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:52.459+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:52.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:52.460+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:53.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:16:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:16:53.038+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 23950 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:16:53.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:52.989+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:53.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:52.989+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:53.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:52.989+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:53.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:52.989+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:53.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:53.117+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:53.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:53.117+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:53.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:53.117+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:53.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:53.117+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:53.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:53.448+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:53.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:53.448+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:53.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:53.448+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:53.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:53.449+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:54.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:54.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:54.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:54.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:54.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:54.120+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:54.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:54.120+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:54.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:53.984+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:54.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:53.984+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:54.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:53.984+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:54.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:53.984+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:54.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:54.433+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:54.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:54.433+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:54.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:54.433+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:54.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:54.434+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:55.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:54.982+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:55.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:54.982+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:55.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:54.982+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:55.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:54.982+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:55.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:55.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:55.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:55.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:55.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:55.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:55.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:55.090+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:55.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:55.392+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:55.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:55.392+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:55.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:55.392+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:55.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:55.392+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:56.203 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:55.942+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:56.203 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:55.942+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:56.203 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:55.942+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:56.203 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:55.943+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:56.203 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:56.114+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:56.204 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:56.114+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:56.204 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:56.114+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:56.204 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:56.114+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:56.502 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:56.362+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:56.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:56.362+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:56.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:56.362+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:56.503 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:56.362+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:57.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:57.134+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:57.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:57.134+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:57.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:57.134+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:57.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:57.134+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:57.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:56.912+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:57.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:56.912+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:57.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:56.912+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:57.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:56.913+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:57.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:57.410+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:57.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:57.410+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:57.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:57.410+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:57.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:57.411+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:58.166 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:16:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:16:58.041+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 24034 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:16:58.167 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:58.130+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:58.167 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:58.130+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:58.167 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:58.130+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:58.167 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:58.131+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:58.167 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:57.914+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:58.168 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:57.914+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:58.168 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:57.914+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:58.168 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:57.914+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:58.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:58.425+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:58.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:58.425+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:58.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:58.425+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:58.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:58.425+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:59.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:58.951+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:59.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:58.951+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:59.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:58.951+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:16:59.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:58.952+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:59.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:59.128+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:59.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:59.128+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:59.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:59.128+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:16:59.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:16:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:16:59.129+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:16:59.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:59.428+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:59.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:59.428+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:59.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:59.428+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:16:59.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:16:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:16:59.428+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:00.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:00.172+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:00.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:00.172+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:00.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:00.172+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:00.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:00.172+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:00.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:59.983+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:00.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:59.983+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:00.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:59.983+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:00.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:16:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:16:59.984+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:00.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:00.467+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:00.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:00.468+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:00.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:00.468+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:00.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:00.468+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:01.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:01.136+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:01.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:01.136+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:01.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:01.136+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:01.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:01.137+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:01.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:00.955+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:01.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:00.955+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:01.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:00.955+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:01.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:00.956+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:01.946 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:01.507+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:01.947 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:01.507+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:01.947 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:01.507+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:01.947 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:01.507+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:02.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:01.946+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:02.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:01.946+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:02.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:01.946+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:02.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:01.947+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:02.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:02.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:02.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:02.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:02.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:02.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:02.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:02.131+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:02.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:02.477+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:02.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:02.477+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:02.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:02.477+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:02.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:02.478+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:03.171 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:17:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:17:03.045+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 24119 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:17:03.172 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:03.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:03.172 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:03.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:03.172 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:03.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:03.172 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:03.098+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:03.172 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:02.910+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:03.173 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:02.910+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:03.173 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:02.910+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:03.173 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:02.911+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:03.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:03.514+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:03.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:03.514+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:03.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:03.514+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:03.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:03.514+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:04.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:03.960+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:04.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:03.960+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:04.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:03.960+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:04.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:03.960+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:04.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:04.068+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:04.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:04.068+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:04.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:04.068+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:04.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:04.069+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:04.998 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:04.506+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:04.998 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:04.506+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:04.998 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:04.506+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:04.999 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:04.506+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:04.997+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:04.997+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:04.997+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:05.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:04.998+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:05.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:05.102+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:05.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:05.102+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:05.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:05.102+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:05.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:05.103+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:05.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:05.492+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:05.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:05.492+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:05.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:05.492+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:05.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:05.492+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:06.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:05.971+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:06.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:05.971+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:06.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:05.971+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:06.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:05.972+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:06.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:06.138+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:06.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:06.138+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:06.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:06.138+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:06.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:06.138+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:06.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:06.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:06.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:06.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:06.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:06.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:06.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:06.454+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:07.454 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:07.016+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:07.454 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:07.016+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:07.454 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:07.016+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:07.454 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:07.016+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:07.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:07.187+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:07.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:07.187+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:07.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:07.187+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:07.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:07.187+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:07.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:07.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:07.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:07.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:07.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:07.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:07.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:07.454+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:08.432 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:17:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:17:08.049+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 24203 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:17:08.432 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:08.146+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:08.433 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:08.146+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:08.433 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:08.146+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:08.433 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:08.147+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:08.433 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:08.055+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:08.433 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:08.055+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:08.433 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:08.055+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:08.433 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:08.056+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:08.430+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:08.430+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:08.430+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:08.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:08.431+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:09.422 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:09.036+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:09.422 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:09.036+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:09.422 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:09.036+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:09.422 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:09.036+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:09.423 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:09.140+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:09.423 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:09.140+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:09.423 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:09.140+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:09.423 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:09.141+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:09.421+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:09.421+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:09.421+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:09.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:09.422+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:10.378 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:10.062+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:10.378 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:10.062+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:10.378 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:10.062+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:10.378 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:10.062+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:10.378 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:10.130+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:10.379 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:10.130+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:10.379 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:10.130+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:10.379 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:10.130+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:10.377+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:10.377+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:10.377+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:10.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:10.377+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:11.373 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:11.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:11.373 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:11.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:11.373 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:11.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:11.373 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:11.113+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:11.373 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:11.043+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:11.373 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:11.043+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:11.374 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:11.043+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:11.374 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:11.043+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:11.727 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:11.372+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:11.728 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:11.372+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:11.728 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:11.372+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:11.728 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:11.372+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:12.370 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:12.007+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:12.370 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:12.007+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:12.370 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:12.007+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:12.370 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:12.008+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:12.370 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:12.092+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:12.371 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:12.092+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:12.371 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:12.092+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:12.371 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:12.092+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:12.691 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:12.369+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:12.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:12.369+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:12.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:12.369+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:12.692 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:12.369+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:13.393 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:17:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:17:13.052+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 24288 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:17:13.394 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:13.039+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:13.394 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:13.039+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:13.394 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:13.039+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:13.394 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:13.040+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:13.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:13.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:13.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:13.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:13.394 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:13.088+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:13.395 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:13.088+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:13.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:13.392+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:13.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:13.392+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:13.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:13.392+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:13.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:13.393+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:14.361 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:14.021+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:14.362 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:14.021+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:14.362 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:14.021+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:14.362 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:14.021+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:14.362 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:14.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:14.362 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:14.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:14.362 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:14.071+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:14.363 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:14.072+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:14.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:14.361+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:14.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:14.361+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:14.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:14.361+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:14.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:14.361+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:15.400 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:15.053+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:15.400 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:15.053+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:15.400 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:15.053+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:15.401 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:15.053+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:15.401 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:15.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:15.401 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:15.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:15.401 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:15.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:15.401 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:15.112+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:15.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:15.399+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:15.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:15.399+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:15.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:15.399+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:15.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:15.400+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:16.447 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:16.044+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:16.447 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:16.044+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:16.447 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:16.044+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:16.447 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:16.044+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:16.447 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:16.072+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:16.448 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:16.072+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:16.448 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:16.072+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:16.448 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:16.072+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:16.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:16.446+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:16.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:16.446+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:16.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:16.446+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:16.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:16.446+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:17.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:17.054+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:17.454 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:17.054+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:17.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:17.054+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:17.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:17.054+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:17.455 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:17.050+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:17.455 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:17.050+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:17.455 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:17.050+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:17.455 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:17.050+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:17.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:17.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:17.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:17.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:17.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:17.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:17.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:17.454+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:18.434 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:17:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:17:18.056+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 24372 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:17:18.435 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:18.069+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:18.435 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:18.069+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:18.435 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:18.069+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:18.435 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:18.070+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:18.436 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:18.096+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:18.436 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:18.096+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:18.436 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:18.096+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:18.436 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:18.097+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:18.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:18.434+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:18.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:18.434+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:18.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:18.434+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:18.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:18.434+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:19.424 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:19.080+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:19.424 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:19.080+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:19.424 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:19.080+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:19.425 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:19.080+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:19.425 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:19.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:19.425 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:19.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:19.425 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:19.133+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:19.425 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:19.133+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:19.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:19.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:19.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:19.424+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:20.463 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:20.118+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:20.463 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:20.118+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:20.463 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:20.118+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:20.463 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:20.118+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:20.464 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:20.103+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:20.464 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:20.103+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:20.464 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:20.103+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:20.464 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:20.103+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:20.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:20.462+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:20.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:20.462+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:20.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:20.462+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:20.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:20.463+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:21.490 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:21.134+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:21.491 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:21.134+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:21.491 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:21.134+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:21.491 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:21.134+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:21.491 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:21.116+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:21.491 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:21.116+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:21.492 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:21.116+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:21.492 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:21.116+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:21.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:21.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:21.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:21.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:21.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:21.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:21.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:21.490+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:22.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:22.136+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:22.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:22.136+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:22.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:22.136+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:22.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:22.137+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:22.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:22.145+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:22.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:22.145+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:22.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:22.145+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:22.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:22.146+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:22.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:22.495+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:22.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:22.495+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:22.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:22.495+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:22.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:22.495+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:23.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:17:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:17:23.060+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 24457 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:17:23.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:23.104+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:23.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:23.104+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:23.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:23.104+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:23.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:23.105+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:23.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:23.116+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:23.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:23.116+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:23.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:23.116+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:23.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:23.117+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:24.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:23.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:24.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:23.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:24.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:23.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:24.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:23.520+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:24.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:24.120+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:24.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:24.120+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:24.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:24.120+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:24.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:24.120+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:24.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:24.146+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:24.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:24.146+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:24.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:24.146+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:24.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:24.146+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:24.980 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:24.552+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:24.980 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:24.552+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:24.981 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:24.552+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:24.981 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:24.553+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:25.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:25.144+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:25.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:25.144+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:25.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:25.144+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:25.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:25.145+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:25.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:25.191+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:25.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:25.191+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:25.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:25.191+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:25.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:25.191+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:25.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:25.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:25.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:25.516+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:26.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:26.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:26.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:26.179+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:26.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:26.179+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:26.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:26.149+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:26.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:26.149+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:26.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:26.149+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:26.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:26.149+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:27.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:26.562+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:27.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:26.562+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:27.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:26.562+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:27.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:26.562+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:27.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:27.113+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:27.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:27.113+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:27.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:27.113+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:27.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:27.113+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:27.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:27.229+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:27.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:27.229+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:27.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:27.229+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:27.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:27.229+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:28.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:27.579+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:28.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:27.579+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:28.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:27.579+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:28.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:27.579+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:28.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:17:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:17:28.064+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 24541 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:17:28.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:28.235+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:28.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:28.235+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:28.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:28.235+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:28.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:28.236+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:28.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:28.129+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:28.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:28.129+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:28.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:28.129+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:28.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:28.129+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:29.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:28.621+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:29.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:28.621+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:29.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:28.621+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:29.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:28.621+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:29.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:29.107+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:29.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:29.107+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:29.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:29.107+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:29.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:29.108+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:29.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:29.272+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:29.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:29.272+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:29.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:29.272+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:29.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:29.272+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:30.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:29.644+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:30.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:29.644+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:30.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:29.644+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:30.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:29.644+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:30.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:30.139+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:30.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:30.139+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:30.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:30.139+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:30.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:30.139+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:30.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:30.224+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:30.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:30.224+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:30.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:30.224+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:30.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:30.224+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:31.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:30.693+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:31.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:30.693+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:31.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:30.693+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:31.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:30.694+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:31.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:31.140+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:31.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:31.140+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:31.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:31.140+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:31.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:31.140+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:31.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:31.216+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:31.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:31.216+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:31.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:31.216+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:31.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:31.216+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:31.681+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:31.681+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:31.681+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:31.681+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:32.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:32.182+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:32.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:32.182+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:32.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:32.182+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:32.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:32.182+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:32.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:32.195+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:32.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:32.195+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:32.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:32.195+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:32.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:32.196+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:32.673+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:32.673+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:32.673+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:33.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:32.673+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:33.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:17:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:17:33.068+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 24626 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:17:33.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:33.225+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:33.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:33.225+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:33.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:33.225+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:33.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:33.225+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:33.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:33.238+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:33.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:33.238+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:33.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:33.238+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:33.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:33.238+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:33.713+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:33.713+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:33.713+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:33.713+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:34.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:34.251+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:34.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:34.251+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:34.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:34.251+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:34.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:34.251+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:34.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:34.227+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:34.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:34.227+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:34.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:34.227+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:34.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:34.228+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:35.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:34.681+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:35.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:34.681+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:35.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:34.681+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:35.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:34.681+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:35.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:35.213+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:35.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:35.213+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:35.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:35.213+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:35.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:35.213+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:35.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:35.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:35.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:35.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:35.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:35.197+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:35.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:35.197+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:35.643+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:35.643+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:35.643+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:35.643+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:36.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:36.198+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:36.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:36.198+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:36.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:36.198+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:36.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:36.198+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:36.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:36.168+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:36.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:36.168+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:36.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:36.168+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:36.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:36.168+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:37.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:36.672+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:37.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:36.673+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:37.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:36.673+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:37.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:36.673+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:37.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:37.168+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:37.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:37.168+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:37.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:37.168+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:37.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:37.169+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:37.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:37.198+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:37.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:37.198+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:37.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:37.198+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:37.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:37.198+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:38.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:37.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:38.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:37.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:37.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:37.637+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:38.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:17:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:17:38.072+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 24710 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:17:38.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:38.130+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:38.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:38.130+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:38.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:38.130+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:38.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:38.130+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:38.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:38.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:38.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:38.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:38.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:38.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:38.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:38.246+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:39.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:38.654+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:39.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:38.654+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:39.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:38.654+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:39.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:38.655+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:39.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:39.111+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:39.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:39.111+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:39.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:39.111+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:39.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:39.111+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:39.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:39.220+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:39.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:39.220+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:39.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:39.220+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:39.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:39.220+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:40.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:39.670+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:40.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:39.670+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:40.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:39.670+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:40.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:39.670+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:40.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:40.150+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:40.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:40.150+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:40.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:40.150+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:40.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:40.150+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:40.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:40.200+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:40.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:40.200+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:40.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:40.200+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:40.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:40.200+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:41.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:40.717+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:41.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:40.717+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:41.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:40.717+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:41.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:40.718+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:41.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:41.120+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:41.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:41.120+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:41.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:41.120+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:41.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:41.120+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:41.228+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:41.228+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:41.228+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:41.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:41.228+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:41.750+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:41.750+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:41.750+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:41.750+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:42.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:42.160+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:42.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:42.160+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:42.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:42.160+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:42.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:42.160+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:42.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:42.231+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:42.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:42.231+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:42.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:42.231+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:42.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:42.231+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:43.077 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:42.789+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:43.077 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:42.789+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:43.077 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:42.789+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:43.077 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:42.789+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:43.411 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:17:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:17:43.075+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 24795 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:17:43.412 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:43.181+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:43.412 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:43.181+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:43.412 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:43.181+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:43.412 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:43.181+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:43.413 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:43.245+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:43.413 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:43.245+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:43.413 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:43.245+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:43.413 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:43.246+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:44.188 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:43.816+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:44.189 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:43.816+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:44.189 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:43.816+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:44.189 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:43.816+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:44.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:44.188+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:44.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:44.188+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:44.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:44.188+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:44.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:44.188+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:44.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:44.252+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:44.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:44.252+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:44.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:44.252+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:44.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:44.252+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:45.161 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:44.846+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:45.162 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:44.846+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:45.162 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:44.846+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:45.162 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:44.846+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:45.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:45.161+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:45.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:45.161+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:45.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:45.161+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:45.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:45.161+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:45.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:45.264+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:45.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:45.264+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:45.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:45.264+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:45.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:45.264+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:46.162 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:45.840+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:46.163 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:45.840+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:46.163 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:45.840+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:46.163 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:45.840+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:46.484 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:46.162+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:46.488 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:46.162+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:46.488 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:46.162+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:46.488 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:46.162+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:46.488 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:46.272+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:46.488 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:46.272+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:46.488 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:46.272+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:46.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:46.272+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:47.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:46.795+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:47.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:46.795+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:47.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:46.795+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:47.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:46.796+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:47.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:47.212+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:47.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:47.212+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:47.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:47.212+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:47.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:47.212+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:47.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:47.268+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:47.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:47.268+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:47.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:47.268+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:47.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:47.269+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:48.080 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:47.818+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:48.081 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:47.818+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:48.081 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:47.818+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:48.081 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:47.818+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:48.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:17:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:17:48.079+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 24879 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:17:48.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:48.221+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:48.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:48.221+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:48.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:48.221+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:48.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:48.221+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:48.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:48.259+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:48.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:48.259+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:48.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:48.259+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:48.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:48.259+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:49.187 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:48.827+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:49.188 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:48.827+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:49.188 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:48.827+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:49.188 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:48.827+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:49.187+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:49.187+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:49.187+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:49.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:49.187+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:49.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:49.289+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:49.505 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:49.289+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:49.505 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:49.289+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:49.505 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:49.289+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:50.181 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:49.788+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:50.182 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:49.788+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:50.182 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:49.788+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:50.182 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:49.789+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:50.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:50.181+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:50.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:50.181+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:50.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:50.181+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:50.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:50.181+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:50.300+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:50.300+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:50.300+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:50.300+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:51.223 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:50.822+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:51.224 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:50.822+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:51.224 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:50.822+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:51.224 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:50.822+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:51.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:51.222+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:51.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:51.222+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:51.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:51.222+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:51.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:51.222+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:51.286+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:51.286+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:51.286+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:51.286+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:51.787+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:51.787+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:51.787+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:51.787+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:52.628 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:52.265+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:52.628 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:52.265+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:52.628 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:52.265+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:52.629 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:52.266+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:52.629 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:52.284+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:52.629 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:52.284+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:52.629 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:52.284+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:52.629 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:52.284+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:53.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:52.815+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:53.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:52.815+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:53.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:52.815+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:53.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:52.815+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:53.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:17:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:17:53.082+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 24964 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:17:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:53.248+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:53.248+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:53.248+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:53.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:53.248+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:53.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:53.302+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:53.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:53.302+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:53.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:53.302+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:53.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:53.303+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:54.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:53.844+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:54.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:53.844+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:54.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:53.844+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:54.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:53.844+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:54.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:54.288+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:54.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:54.288+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:54.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:54.288+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:54.288+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:54.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:54.224+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:54.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:54.224+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:54.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:54.224+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:54.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:54.224+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:55.233 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:54.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:55.234 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:54.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:55.234 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:54.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:55.234 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:54.802+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:55.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:55.238+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:55.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:55.238+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:55.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:55.238+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:55.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:55.238+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:55.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:55.232+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:55.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:55.232+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:55.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:55.232+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:55.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:55.232+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:56.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:55.770+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:56.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:55.770+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:56.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:55.770+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:55.770+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:56.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:56.262+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:56.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:56.262+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:56.262+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:56.262+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:56.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:56.259+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:56.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:56.259+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:56.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:56.259+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:56.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:56.259+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:57.242 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:56.803+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:57.242 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:56.803+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:57.242 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:56.803+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:57.242 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:56.803+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:57.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:57.259+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:57.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:57.259+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:57.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:57.259+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:57.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:57.259+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:57.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:57.240+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:57.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:57.240+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:57.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:57.240+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:57.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:57.240+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:58.088 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:57.782+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:58.088 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:57.782+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:58.088 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:57.782+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:58.088 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:57.782+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:58.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:17:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:17:58.086+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 25048 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:17:58.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:58.289+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:58.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:58.289+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:58.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:58.289+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:58.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:58.289+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:58.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:58.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:58.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:58.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:58.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:58.206+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:58.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:58.206+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:58.741+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:58.741+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:58.741+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:17:59.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:58.741+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:59.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:59.261+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:59.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:59.262+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:59.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:59.262+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:17:59.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:17:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:17:59.262+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:17:59.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:59.228+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:59.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:59.228+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:59.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:59.228+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:17:59.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:17:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:17:59.228+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:00.183 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:59.784+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:00.183 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:59.784+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:00.183 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:59.784+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:00.184 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:17:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:17:59.785+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:00.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:00.255+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:00.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:00.255+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:00.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:00.255+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:00.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:00.255+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:00.182+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:00.182+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:00.182+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:00.182+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:01.188 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:00.783+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:01.188 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:00.783+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:01.188 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:00.783+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:01.188 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:00.783+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:01.187+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:01.187+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:01.187+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:01.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:01.187+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:01.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:01.216+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:01.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:01.216+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:01.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:01.216+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:01.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:01.216+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:01.828+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:01.828+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:02.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:01.828+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:02.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:01.828+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:02.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:02.213+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:02.505 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:02.213+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:02.505 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:02.213+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:02.505 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:02.213+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:02.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:02.231+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:02.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:02.231+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:02.506 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:02.231+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:02.506 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:02.231+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:03.203 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:18:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:18:03.090+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 25133 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:18:03.204 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:02.878+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:03.204 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:02.879+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:03.204 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:02.879+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:03.204 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:02.879+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:03.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:03.202+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:03.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:03.202+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:03.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:03.202+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:03.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:03.202+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:03.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:03.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:03.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:03.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:03.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:03.215+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:03.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:03.215+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:03.850+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:03.850+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:04.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:03.850+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:04.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:03.850+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:04.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:04.244+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:04.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:04.244+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:04.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:04.244+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:04.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:04.244+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:04.170+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:04.170+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:04.170+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:04.171+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:05.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:04.843+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:05.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:04.843+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:05.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:04.843+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:05.214 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:04.843+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:05.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:05.277+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:05.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:05.277+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:05.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:05.277+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:05.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:05.277+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:05.212+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:05.212+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:05.212+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:05.212+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:06.247 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:05.864+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:06.248 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:05.864+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:06.248 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:05.864+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:06.248 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:05.864+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:06.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:06.320+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:06.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:06.320+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:06.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:06.320+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:06.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:06.320+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:06.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:06.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:06.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:06.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:06.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:06.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:06.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:06.246+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:07.210 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:06.879+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:07.210 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:06.879+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:07.210 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:06.879+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:07.211 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:06.880+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:07.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:07.330+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:07.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:07.330+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:07.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:07.330+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:07.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:07.331+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:07.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:07.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:07.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:07.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:07.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:07.209+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:07.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:07.210+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:08.221 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:18:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:18:08.095+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 25217 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:18:08.222 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:07.915+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:08.222 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:07.915+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:08.222 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:07.915+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:08.222 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:07.915+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:08.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:08.330+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:08.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:08.330+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:08.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:08.330+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:08.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:08.331+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:08.234+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:08.234+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:08.234+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:08.234+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:09.189 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:08.912+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:09.190 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:08.912+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:09.190 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:08.912+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:09.190 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:08.912+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:09.188+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:09.188+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:09.188+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:09.189+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:09.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:09.346+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:09.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:09.346+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:09.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:09.346+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:09.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:09.346+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:10.203 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:09.879+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:10.203 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:09.879+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:10.203 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:09.879+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:10.203 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:09.880+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:10.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:10.378+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:10.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:10.378+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:10.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:10.378+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:10.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:10.378+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:10.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:10.202+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:10.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:10.202+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:10.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:10.202+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:10.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:10.202+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:11.243 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:10.908+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:11.243 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:10.908+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:11.243 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:10.908+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:11.243 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:10.908+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:11.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:11.391+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:11.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:11.391+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:11.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:11.391+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:11.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:11.391+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:11.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:11.242+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:11.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:11.242+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:11.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:11.242+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:11.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:11.242+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:12.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:11.929+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:12.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:11.929+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:12.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:11.929+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:11.930+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:12.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:12.346+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:12.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:12.346+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:12.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:12.346+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:12.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:12.346+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:12.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:12.226+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:12.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:12.226+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:12.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:12.226+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:12.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:12.226+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:13.227 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:18:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:18:13.099+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 25302 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:18:13.228 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:12.921+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:13.228 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:12.921+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:13.228 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:12.921+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:13.229 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:12.922+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:13.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:13.316+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:13.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:13.316+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:13.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:13.316+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:13.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:13.317+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:13.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:13.248+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:13.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:13.248+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:13.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:13.248+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:13.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:13.248+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:14.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:13.936+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:14.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:13.936+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:14.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:13.936+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:14.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:13.937+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:14.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:14.297+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:14.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:14.297+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:14.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:14.297+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:14.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:14.297+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:14.335+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:14.335+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:14.335+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:14.336+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:14.901+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:14.901+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:15.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:14.901+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:15.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:14.901+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:15.319+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:15.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:15.319+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:15.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:15.319+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:15.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:15.320+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:15.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:15.332+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:15.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:15.332+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:15.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:15.332+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:15.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:15.332+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:15.951+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:15.951+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:15.951+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:16.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:15.952+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:16.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:16.310+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:16.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:16.310+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:16.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:16.310+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:16.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:16.310+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:16.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:16.343+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:16.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:16.343+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:16.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:16.343+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:16.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:16.344+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:17.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:16.915+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:17.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:16.915+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:17.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:16.915+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:17.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:16.915+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:17.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:17.349+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:17.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:17.349+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:17.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:17.349+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:17.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:17.349+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:17.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:17.347+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:17.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:17.347+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:17.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:17.347+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:17.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:17.348+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:18.228 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:18:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:18:18.103+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 25386 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:18:18.228 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:17.894+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:18.229 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:17.894+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:18.229 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:17.894+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:18.229 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:17.894+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:18.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:18.383+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:18.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:18.383+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:18.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:18.383+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:18.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:18.383+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:18.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:18.329+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:18.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:18.329+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:18.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:18.329+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:18.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:18.330+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:19.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:18.937+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:19.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:18.937+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:19.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:18.937+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:19.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:18.937+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:19.379+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:19.379+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:19.379+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:19.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:19.379+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:19.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:19.318+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:19.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:19.318+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:19.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:19.318+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:19.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:19.318+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:20.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:19.914+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:20.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:19.914+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:20.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:19.914+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:20.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:19.915+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:20.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:20.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:20.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:20.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:20.356+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:20.303+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:20.303+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:20.303+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:20.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:20.303+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:21.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:20.957+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:21.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:20.957+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:21.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:20.957+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:20.958+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:21.733 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:21.392+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:21.734 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:21.392+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:21.734 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:21.392+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:21.734 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:21.393+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:21.734 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:21.352+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:21.734 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:21.352+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:21.734 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:21.352+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:21.734 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:21.353+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:21.978+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:21.978+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:21.978+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:21.978+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:22.304+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:22.304+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:22.304+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:22.304+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:22.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:22.421+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:22.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:22.421+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:22.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:22.421+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:22.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:22.421+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:23.236 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:18:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:18:23.107+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 25471 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:18:23.237 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:22.956+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:23.237 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:22.956+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:23.237 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:22.956+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:23.237 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:22.957+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:23.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:23.292+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:23.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:23.292+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:23.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:23.292+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:23.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:23.292+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:23.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:23.456+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:23.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:23.456+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:23.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:23.456+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:23.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:23.457+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:23.932+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:24.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:23.932+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:24.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:23.932+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:24.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:23.932+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:24.259+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:24.259+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:24.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:24.259+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:24.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:24.260+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:24.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:24.480+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:24.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:24.480+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:24.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:24.480+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:24.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:24.480+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:25.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:24.888+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:25.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:24.888+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:25.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:24.888+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:25.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:24.888+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:25.254+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:25.254+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:25.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:25.254+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:25.255+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:25.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:25.489+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:25.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:25.489+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:25.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:25.489+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:25.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:25.489+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:26.234 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:25.857+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:26.235 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:25.857+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:26.235 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:25.857+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:26.235 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:25.857+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:26.500 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:26.234+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:26.500 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:26.234+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:26.500 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:26.234+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:26.500 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:26.234+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:26.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:26.499+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:26.499+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:26.499+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:26.499+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:27.188 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:26.817+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:27.189 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:26.817+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:27.189 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:26.817+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:27.190 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:26.817+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:27.496 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:27.188+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:27.496 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:27.188+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:27.496 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:27.188+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:27.496 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:27.188+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:27.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:27.495+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:27.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:27.495+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:27.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:27.495+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:27.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:27.495+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:28.112 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:27.825+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:28.112 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:27.825+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:28.113 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:27.825+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:28.113 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:27.826+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:28.458 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:18:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:18:28.111+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 25555 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:18:28.458 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:28.152+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:28.459 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:28.152+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:28.459 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:28.152+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:28.459 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:28.152+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:28.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:28.457+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:28.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:28.457+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:28.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:28.457+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:28.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:28.458+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:29.200 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:28.778+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:29.200 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:28.778+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:29.200 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:28.778+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:29.201 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:28.778+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:29.484 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:29.199+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:29.485 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:29.199+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:29.485 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:29.199+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:29.485 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:29.199+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:29.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:29.483+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:29.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:29.483+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:29.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:29.483+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:29.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:29.484+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:30.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:29.749+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:30.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:29.749+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:30.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:29.749+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:30.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:29.749+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:30.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:30.189+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:30.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:30.189+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:30.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:30.189+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:30.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:30.190+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:30.800 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:30.532+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:30.800 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:30.532+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:30.800 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:30.532+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:30.800 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:30.532+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:31.231 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:30.799+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:31.232 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:30.799+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:31.232 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:30.799+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:31.232 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:30.799+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:31.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:31.230+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:31.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:31.230+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:31.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:31.230+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:31.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:31.231+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:31.837 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:31.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:31.837 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:31.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:31.837 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:31.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:31.838 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:31.508+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:31.836+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:31.836+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:31.836+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:32.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:31.837+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:32.564 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:32.266+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:32.564 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:32.266+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:32.564 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:32.266+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:32.565 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:32.266+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:32.565 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:32.505+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:32.565 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:32.505+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:32.565 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:32.505+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:32.565 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:32.505+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:33.116 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:32.831+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:33.116 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:32.831+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:33.116 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:32.831+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:33.116 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:32.831+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:33.472 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:18:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:18:33.115+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 25639 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:18:33.473 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:33.267+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:33.473 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:33.267+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:33.473 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:33.267+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:33.473 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:33.267+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:33.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:33.472+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:33.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:33.472+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:33.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:33.472+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:33.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:33.472+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:33.819+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:33.819+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:33.819+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:34.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:33.819+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:34.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:34.482+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:34.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:34.482+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:34.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:34.482+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:34.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:34.483+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:34.301+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:34.301+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:34.301+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:34.301+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:35.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:34.815+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:35.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:34.815+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:35.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:34.815+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:35.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:34.815+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:35.636 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:35.488+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:35.636 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:35.488+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:35.636 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:35.488+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:35.637 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:35.488+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:35.637 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:35.267+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:35.637 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:35.267+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:35.637 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:35.267+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:35.637 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:35.267+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:35.796+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:35.796+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:35.796+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:36.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:35.796+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:36.525 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:36.254+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:36.525 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:36.254+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:36.525 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:36.254+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:36.526 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:36.254+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:36.804 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:36.524+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:36.804 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:36.524+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:36.804 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:36.524+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:36.804 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:36.524+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:37.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:36.803+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:37.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:36.803+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:37.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:36.803+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:37.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:36.804+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:37.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:37.512+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:37.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:37.512+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:37.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:37.512+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:37.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:37.512+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:37.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:37.275+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:37.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:37.275+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:37.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:37.275+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:37.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:37.275+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:38.120 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:37.821+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:38.121 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:37.821+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:38.121 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:37.821+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:38.121 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:37.822+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:38.480 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:18:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:18:38.119+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 25722 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:18:38.480 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:38.284+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:38.480 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:38.284+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:38.481 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:38.284+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:38.481 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:38.284+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:38.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:38.478+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:38.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:38.478+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:38.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:38.478+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:38.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:38.479+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:39.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:38.838+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:39.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:38.838+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:39.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:38.838+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:39.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:38.839+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:39.263+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:39.263+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:39.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:39.263+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:39.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:39.263+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:39.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:39.473+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:39.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:39.473+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:39.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:39.473+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:39.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:39.473+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:40.220 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:39.867+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:40.220 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:39.867+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:40.220 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:39.867+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:40.221 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:39.867+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:40.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:40.425+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:40.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:40.425+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:40.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:40.425+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:40.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:40.425+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:40.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:40.219+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:40.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:40.219+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:40.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:40.219+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:40.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:40.220+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:41.249 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:40.883+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:41.249 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:40.883+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:41.250 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:40.883+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:41.250 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:40.883+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:41.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:41.248+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:41.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:41.248+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:41.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:41.248+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:41.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:41.249+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:41.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:41.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:41.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:41.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:41.405+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:41.864+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:41.864+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:42.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:41.864+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:42.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:41.864+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:42.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:42.203+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:42.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:42.203+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:42.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:42.203+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:42.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:42.203+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:42.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:42.418+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:42.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:42.418+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:42.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:42.418+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:42.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:42.418+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:43.137 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:18:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:18:43.123+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 25806 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:18:43.138 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:42.877+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:43.138 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:42.877+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:43.138 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:42.877+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:43.138 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:42.877+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:43.447 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:43.207+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:43.448 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:43.207+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:43.448 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:43.207+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:43.448 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:43.207+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:43.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:43.446+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:43.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:43.446+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:43.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:43.446+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:43.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:43.447+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:44.183 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:43.847+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:44.184 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:43.847+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:44.184 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:43.847+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:44.184 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:43.848+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:44.449 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:44.182+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:44.450 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:44.182+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:44.450 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:44.182+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:44.450 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:44.183+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:44.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:44.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:44.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:44.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:44.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:44.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:44.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:44.449+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:45.180 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:44.832+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:45.181 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:44.832+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:45.181 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:44.832+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:45.181 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:44.833+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:45.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:45.179+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:45.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:45.179+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:45.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:45.179+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:45.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:45.180+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:45.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:45.406+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:45.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:45.406+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:45.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:45.406+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:45.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:45.406+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:46.137 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:45.861+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:46.137 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:45.861+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:46.137 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:45.861+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:46.137 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:45.861+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:46.441 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:46.136+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:46.441 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:46.136+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:46.442 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:46.136+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:46.442 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:46.136+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:46.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:46.440+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:46.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:46.440+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:46.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:46.440+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:46.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:46.441+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:47.160 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:46.820+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:47.160 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:46.820+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:47.160 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:46.820+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:47.161 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:46.820+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:47.486 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:47.159+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:47.486 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:47.159+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:47.486 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:47.159+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:47.486 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:47.160+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:47.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:47.485+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:47.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:47.485+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:47.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:47.485+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:47.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:47.485+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:48.122 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:47.797+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:48.123 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:47.797+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:48.123 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:47.797+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:48.123 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:47.798+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:48.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:18:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:18:48.127+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 25890 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:18:48.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:48.121+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:48.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:48.121+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:48.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:48.121+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:48.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:48.122+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:49.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:48.528+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:49.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:48.528+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:49.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:48.528+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:49.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:48.528+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:49.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:48.767+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:49.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:48.767+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:49.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:48.767+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:49.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:48.768+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:49.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:49.147+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:49.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:49.147+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:49.147+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:49.147+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:50.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:49.523+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:50.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:49.523+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:50.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:49.523+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:50.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:49.523+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:50.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:49.762+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:50.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:49.762+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:50.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:49.762+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:50.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:49.762+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:50.486 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:50.131+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:50.487 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:50.131+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:50.487 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:50.131+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:50.487 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:50.132+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:50.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:50.486+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:50.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:50.486+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:50.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:50.486+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:50.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:50.486+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:51.172 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:50.752+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:51.173 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:50.752+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:51.173 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:50.752+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:51.173 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:50.753+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:51.479 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:51.172+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:51.479 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:51.172+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:51.479 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:51.172+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:51.479 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:51.172+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:51.737 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:51.478+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:51.738 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:51.478+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:51.738 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:51.478+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:51.738 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:51.478+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:51.773+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:51.773+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:51.773+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:51.773+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:52.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:52.218+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:52.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:52.218+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:52.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:52.218+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:52.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:52.218+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:52.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:52.462+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:52.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:52.462+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:52.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:52.462+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:52.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:52.462+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:53.132 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:52.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:53.132 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:52.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:53.132 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:52.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:53.133 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:52.802+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:53.421 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:18:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:18:53.131+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 25975 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:18:53.422 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:53.237+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:53.422 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:53.237+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:53.422 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:53.237+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:53.422 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:53.237+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:53.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:53.420+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:53.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:53.420+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:53.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:53.420+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:53.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:53.421+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:54.068 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:53.811+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:54.068 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:53.811+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:54.068 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:53.811+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:54.068 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:53.811+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:54.451 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:54.199+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:54.451 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:54.199+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:54.451 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:54.199+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:54.451 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:54.200+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:54.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:54.450+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:54.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:54.450+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:54.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:54.450+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:54.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:54.450+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:55.211 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:54.781+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:55.211 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:54.781+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:55.211 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:54.781+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:55.212 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:54.782+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:55.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:55.210+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:55.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:55.210+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:55.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:55.210+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:55.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:55.210+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:55.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:55.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:55.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:55.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:55.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:55.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:55.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:55.405+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:56.184 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:55.777+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:56.185 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:55.777+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:56.185 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:55.777+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:56.185 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:55.777+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:56.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:56.364+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:56.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:56.364+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:56.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:56.364+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:56.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:56.364+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:56.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:56.183+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:56.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:56.183+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:56.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:56.183+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:56.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:56.184+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:57.146 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:56.778+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:57.146 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:56.778+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:57.146 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:56.778+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:57.147 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:56.779+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:57.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:57.145+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:57.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:57.145+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:57.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:57.145+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:57.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:57.145+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:57.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:57.390+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:57.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:57.390+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:57.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:57.390+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:57.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:57.390+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:58.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:57.744+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:58.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:57.744+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:58.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:57.744+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:58.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:57.744+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:58.396 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:18:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:18:58.135+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 26059 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:18:58.396 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:58.099+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:58.397 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:58.099+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:58.397 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:58.099+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:58.397 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:58.099+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:58.744 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:58.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:58.745 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:58.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:58.745 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:58.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:58.745 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:58.396+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:58.744+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:59.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:58.744+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:59.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:58.744+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:18:59.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:58.744+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:59.352 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:59.079+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:59.352 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:59.079+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:59.352 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:59.079+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:18:59.352 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:18:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:18:59.079+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:18:59.736 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:59.351+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:59.737 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:59.351+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:59.737 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:59.351+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:18:59.737 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:18:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:18:59.351+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:00.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:59.736+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:00.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:59.736+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:00.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:59.736+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:00.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:18:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:18:59.736+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:00.387 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:00.123+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:00.387 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:00.123+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:00.387 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:00.123+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:00.387 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:00.123+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:00.723 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:00.386+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:00.723 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:00.386+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:00.723 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:00.386+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:00.723 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:00.386+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:01.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:00.722+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:00.722+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:00.722+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:00.722+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:01.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:01.171+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:01.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:01.171+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:01.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:01.171+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:01.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:01.171+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:01.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:01.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:01.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:01.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:01.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:01.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:01.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:01.405+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:02.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:01.704+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:02.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:01.704+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:02.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:01.704+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:02.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:01.704+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:02.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:02.168+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:02.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:02.168+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:02.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:02.168+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:02.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:02.168+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:02.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:02.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:02.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:02.396+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:03.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:02.696+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:03.007 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:02.696+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:03.007 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:02.696+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:03.007 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:02.697+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:03.443 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:19:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:19:03.139+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 26144 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:19:03.443 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:03.218+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:03.443 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:03.218+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:03.443 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:03.218+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:03.443 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:03.218+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:03.699 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:03.442+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:03.699 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:03.442+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:03.699 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:03.442+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:03.699 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:03.442+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:03.697+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:03.697+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:04.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:03.697+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:04.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:03.698+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:04.658 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:04.257+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:04.658 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:04.257+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:04.659 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:04.257+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:04.659 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:04.258+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:04.659 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:04.400+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:04.659 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:04.400+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:04.659 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:04.400+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:04.659 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:04.400+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:05.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:04.657+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:05.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:04.657+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:05.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:04.657+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:05.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:04.657+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:05.654 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:05.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:05.655 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:05.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:05.655 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:05.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:05.655 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:05.357+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:05.655 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:05.278+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:05.655 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:05.278+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:05.656 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:05.278+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:05.656 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:05.278+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:06.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:05.654+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:06.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:05.654+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:06.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:05.654+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:06.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:05.654+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:06.667 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:06.359+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:06.667 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:06.359+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:06.667 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:06.359+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:06.667 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:06.359+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:06.668 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:06.306+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:06.668 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:06.306+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:06.668 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:06.306+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:06.668 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:06.306+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:07.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:06.666+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:07.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:06.666+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:07.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:06.666+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:07.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:06.666+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:07.648 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:07.351+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:07.648 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:07.351+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:07.648 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:07.351+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:07.648 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:07.351+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:07.649 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:07.329+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:07.649 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:07.329+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:07.649 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:07.329+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:07.649 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:07.329+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:08.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:07.648+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:08.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:07.648+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:08.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:07.648+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:08.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:07.648+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:08.269 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:19:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:19:08.143+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 26228 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:19:08.637 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:08.371+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:08.638 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:08.371+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:08.638 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:08.371+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:08.638 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:08.371+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:08.638 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:08.319+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:08.638 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:08.319+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:08.638 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:08.319+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:08.638 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:08.319+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:09.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:08.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:09.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:08.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:09.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:08.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:09.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:08.637+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:09.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:09.334+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:09.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:09.334+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:09.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:09.334+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:09.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:09.334+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:09.633 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:09.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:09.633 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:09.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:09.633 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:09.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:09.633 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:09.390+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:10.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:09.632+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:10.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:09.632+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:10.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:09.632+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:10.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:09.632+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:10.602 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:10.324+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:10.602 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:10.324+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:10.602 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:10.324+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:10.602 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:10.324+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:10.603 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:10.387+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:10.603 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:10.387+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:10.603 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:10.387+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:10.603 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:10.387+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:11.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:10.600+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:11.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:10.600+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:11.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:10.600+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:11.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:10.601+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:11.641 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:11.279+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:11.642 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:11.279+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:11.642 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:11.279+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:11.642 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:11.280+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:11.642 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:11.407+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:11.642 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:11.407+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:11.643 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:11.407+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:11.643 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:11.407+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:12.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:11.641+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:12.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:11.641+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:12.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:11.641+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:11.641+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:12.500 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:12.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:12.500 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:12.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:12.501 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:12.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:12.501 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:12.431+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:12.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:12.239+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:12.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:12.239+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:12.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:12.239+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:12.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:12.239+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:12.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:12.597+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:12.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:12.597+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:12.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:12.597+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:12.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:12.597+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:13.470 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:19:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:19:13.146+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 26312 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:19:13.470 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:13.191+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:13.471 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:13.191+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:13.471 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:13.191+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:13.471 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:13.192+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:13.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:13.605+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:13.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:13.605+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:13.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:13.605+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:13.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:13.605+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:13.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:13.468+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:13.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:13.468+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:13.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:13.468+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:13.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:13.468+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:14.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:14.435+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:14.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:14.435+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:14.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:14.435+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:14.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:14.435+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:14.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:14.231+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:14.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:14.231+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:14.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:14.231+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:14.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:14.232+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:15.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:14.602+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:15.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:14.602+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:15.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:14.602+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:15.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:14.602+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:15.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:15.415+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:15.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:15.415+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:15.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:15.415+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:15.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:15.415+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:15.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:15.195+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:15.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:15.195+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:15.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:15.195+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:15.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:15.195+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:16.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:15.612+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:15.612+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:15.612+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:15.612+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:16.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:16.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:16.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:16.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:16.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:16.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:16.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:16.423+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:16.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:16.221+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:16.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:16.221+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:16.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:16.221+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:16.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:16.221+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:17.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:16.607+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:17.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:16.607+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:17.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:16.607+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:17.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:16.607+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:17.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:17.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:17.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:17.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:17.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:17.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:17.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:17.412+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:17.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:17.217+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:17.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:17.217+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:17.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:17.217+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:17.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:17.217+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:18.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:17.635+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:18.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:17.635+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:18.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:17.635+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:18.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:17.635+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:18.277 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:19:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:19:18.150+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 26396 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:19:18.277 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:18.167+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:18.277 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:18.167+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:18.277 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:18.167+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:18.278 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:18.167+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:18.666 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:18.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:18.667 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:18.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:18.667 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:18.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:18.667 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:18.429+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:19.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:18.664+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:19.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:18.664+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:19.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:18.664+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:19.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:18.665+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:19.412 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:19.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:19.412 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:19.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:19.413 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:19.151+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:19.413 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:19.151+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:19.410+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:19.410+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:19.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:19.410+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:19.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:19.410+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:19.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:19.626+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:19.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:19.626+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:19.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:19.626+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:19.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:19.627+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:20.438 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:20.156+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:20.438 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:20.156+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:20.439 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:20.156+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:20.439 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:20.157+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:20.437+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:20.437+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:20.437+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:20.437+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:20.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:20.633+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:20.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:20.633+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:20.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:20.633+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:20.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:20.633+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:21.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:21.431+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:21.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:21.431+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:21.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:21.431+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:21.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:21.432+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:21.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:21.194+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:21.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:21.194+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:21.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:21.194+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:21.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:21.194+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:22.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:21.618+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:22.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:21.618+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:22.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:21.618+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:22.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:21.618+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:22.456 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:22.186+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:22.456 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:22.186+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:22.456 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:22.186+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:22.456 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:22.186+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:22.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:22.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:22.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:22.454+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:22.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:22.631+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:22.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:22.631+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:22.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:22.631+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:22.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:22.631+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:23.409 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:19:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:19:23.154+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 26481 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:19:23.410 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:23.232+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:23.410 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:23.232+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:23.410 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:23.232+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:23.410 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:23.232+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:23.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:23.622+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:23.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:23.622+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:23.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:23.622+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:23.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:23.622+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:23.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:23.408+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:23.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:23.408+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:23.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:23.408+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:23.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:23.408+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:24.577 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:24.265+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:24.578 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:24.265+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:24.578 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:24.265+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:24.578 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:24.265+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:24.578 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:24.447+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:24.578 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:24.447+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:24.578 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:24.447+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:24.578 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:24.447+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:25.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:24.575+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:25.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:24.575+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:25.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:24.575+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:25.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:24.576+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:25.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:25.460+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:25.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:25.460+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:25.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:25.460+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:25.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:25.460+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:25.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:25.232+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:25.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:25.232+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:25.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:25.232+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:25.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:25.232+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:25.555+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:26.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:25.555+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:26.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:25.555+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:26.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:25.555+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:26.474 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:26.199+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:26.474 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:26.199+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:26.474 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:26.199+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:26.475 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:26.199+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:26.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:26.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:26.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:26.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:26.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:26.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:26.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:26.473+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:26.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:26.521+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:26.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:26.521+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:26.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:26.521+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:26.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:26.521+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:27.491 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:27.160+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:27.492 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:27.160+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:27.492 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:27.160+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:27.492 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:27.160+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:27.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:27.507+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:27.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:27.507+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:27.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:27.507+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:27.507+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:27.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:27.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:27.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:27.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:27.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:27.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:27.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:27.490+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:28.288 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:19:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:19:28.158+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 26565 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:19:28.289 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:28.168+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:28.289 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:28.168+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:28.289 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:28.168+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:28.289 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:28.169+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:28.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:28.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:28.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:28.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:28.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:28.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:28.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:28.451+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:28.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:28.475+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:28.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:28.475+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:28.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:28.475+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:28.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:28.476+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:29.444 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:29.154+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:29.444 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:29.154+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:29.444 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:29.154+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:29.444 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:29.155+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:29.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:29.442+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:29.442+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:29.442+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:29.443+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:29.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:29.489+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:29.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:29.489+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:29.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:29.489+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:29.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:29.489+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:30.423 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:30.166+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:30.423 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:30.166+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:30.423 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:30.166+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:30.423 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:30.166+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:30.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:30.421+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:30.421+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:30.421+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:30.422+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:30.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:30.466+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:30.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:30.466+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:30.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:30.466+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:30.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:30.466+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:31.470 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:31.379+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:31.470 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:31.379+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:31.470 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:31.379+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:31.470 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:31.379+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:31.471 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:31.134+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:31.471 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:31.134+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:31.471 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:31.134+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:31.471 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:31.134+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:31.740 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:31.469+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:31.741 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:31.469+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:31.741 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:31.469+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:31.741 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:31.469+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:32.399 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:32.117+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:32.399 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:32.117+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:32.400 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:32.117+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:32.400 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:32.117+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:32.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:32.513+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:32.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:32.513+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:32.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:32.513+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:32.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:32.513+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:32.398+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:32.398+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:32.398+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:32.399+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:33.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:19:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:19:33.164+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 26650 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:19:33.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:33.350+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:33.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:33.350+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:33.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:33.350+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:33.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:33.350+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:33.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:33.125+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:33.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:33.125+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:33.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:33.125+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:33.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:33.126+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:34.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:33.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:34.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:33.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:34.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:33.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:34.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:33.564+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:34.377 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:34.122+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:34.377 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:34.122+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:34.377 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:34.122+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:34.377 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:34.122+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:34.376+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:34.376+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:34.376+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:34.376+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:34.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:34.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:34.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:34.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:34.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:34.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:34.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:34.521+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:35.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:35.326+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:35.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:35.326+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:35.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:35.326+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:35.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:35.327+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:35.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:35.125+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:35.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:35.125+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:35.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:35.125+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:35.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:35.125+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:35.539+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:35.539+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:36.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:35.539+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:36.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:35.540+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:36.363 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:36.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:36.363 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:36.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:36.364 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:36.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:36.364 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:36.098+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:36.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:36.557+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:36.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:36.557+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:36.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:36.557+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:36.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:36.558+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:36.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:36.362+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:36.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:36.362+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:36.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:36.362+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:36.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:36.363+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:37.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:37.352+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:37.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:37.352+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:37.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:37.352+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:37.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:37.352+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:37.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:37.127+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:37.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:37.127+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:37.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:37.127+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:37.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:37.127+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:38.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:37.552+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:37.552+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:37.552+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:37.552+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:38.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:19:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:19:38.168+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 26734 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:19:38.295 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:38.126+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:38.295 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:38.126+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:38.295 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:38.126+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:38.295 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:38.126+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:38.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:38.524+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:38.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:38.524+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:38.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:38.524+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:38.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:38.524+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:38.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:38.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:38.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:38.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:38.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:38.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:38.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:38.391+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:39.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:39.372+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:39.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:39.372+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:39.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:39.372+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:39.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:39.372+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:39.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:39.152+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:39.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:39.152+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:39.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:39.152+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:39.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:39.152+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:40.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:39.555+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:40.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:39.555+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:40.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:39.555+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:40.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:39.556+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:40.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:40.358+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:40.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:40.358+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:40.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:40.358+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:40.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:40.359+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:40.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:40.150+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:40.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:40.150+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:40.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:40.150+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:40.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:40.150+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:41.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:40.508+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:41.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:40.508+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:41.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:40.508+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:41.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:40.508+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:41.473 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:41.310+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:41.473 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:41.310+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:41.473 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:41.310+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:41.474 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:41.310+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:41.474 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:41.122+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:41.474 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:41.122+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:41.474 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:41.122+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:41.474 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:41.123+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:41.741 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:41.472+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:41.741 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:41.472+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:41.741 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:41.472+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:41.742 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:41.472+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:42.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:42.286+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:42.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:42.286+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:42.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:42.286+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:42.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:42.286+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:42.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:42.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:42.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:42.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:42.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:42.074+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:42.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:42.075+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:43.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:42.518+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:43.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:42.518+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:43.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:42.518+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:43.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:42.519+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:43.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:19:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:19:43.172+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 26819 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:19:43.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:43.290+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:43.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:43.290+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:43.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:43.290+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:43.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:43.291+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:43.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:43.078+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:43.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:43.078+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:43.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:43.078+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:43.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:43.078+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:44.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:43.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:44.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:43.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:44.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:43.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:44.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:43.563+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:44.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:44.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:44.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:44.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:44.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:44.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:44.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:44.339+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:44.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:44.103+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:44.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:44.103+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:44.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:44.103+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:44.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:44.104+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:45.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:44.553+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:44.553+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:44.553+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:44.554+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:45.371 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:45.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:45.372 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:45.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:45.372 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:45.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:45.372 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:45.089+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:45.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:45.595+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:45.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:45.595+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:45.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:45.595+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:45.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:45.596+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:45.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:45.370+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:45.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:45.370+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:45.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:45.370+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:45.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:45.371+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:46.414 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:46.110+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:46.414 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:46.110+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:46.415 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:46.110+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:46.415 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:46.110+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:46.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:46.618+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:46.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:46.618+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:46.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:46.618+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:46.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:46.619+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:46.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:46.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:46.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:46.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:46.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:46.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:46.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:46.413+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:47.430 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:47.096+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:47.431 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:47.096+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:47.431 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:47.096+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:47.431 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:47.096+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:47.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:47.578+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:47.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:47.578+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:47.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:47.578+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:47.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:47.579+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:47.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:47.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:47.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:47.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:47.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:47.429+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:47.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:47.430+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:48.300 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:19:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:19:48.176+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 26930 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:19:48.301 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:48.105+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:48.301 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:48.105+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:48.301 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:48.105+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:48.301 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:48.105+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:48.579 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:48.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:48.579 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:48.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:48.579 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:48.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:48.579 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:48.474+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:49.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:48.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:49.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:48.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:49.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:48.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:49.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:48.578+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:49.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:49.081+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:49.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:49.081+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:49.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:49.081+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:49.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:49.081+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:50.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:49.611+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:50.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:49.611+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:50.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:49.611+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:50.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:49.612+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:50.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:49.524+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:50.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:49.524+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:50.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:49.524+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:50.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:49.524+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:50.123+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:50.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:50.123+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:50.123+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:50.123+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:51.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:50.590+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:51.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:50.590+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:51.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:50.590+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:51.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:50.590+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:51.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:50.565+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:51.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:50.565+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:51.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:50.565+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:51.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:50.565+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:51.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:51.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:51.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:51.089+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:51.089+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:51.546+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:51.546+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:51.546+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:51.547+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:52.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:51.534+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:52.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:51.534+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:52.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:51.534+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:52.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:51.534+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:52.436 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:52.111+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:52.436 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:52.111+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:52.436 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:52.111+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:52.436 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:52.111+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:52.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:52.514+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:52.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:52.514+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:52.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:52.514+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:52.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:52.514+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:52.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:52.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:52.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:52.512+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:53.501 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:19:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:19:53.180+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 27015 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:19:53.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:53.083+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:53.083+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:53.083+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:53.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:53.084+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:53.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:53.500+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:53.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:53.500+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:53.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:53.500+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:53.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:53.501+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:53.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:53.525+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:53.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:53.525+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:53.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:53.525+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:53.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:53.525+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:54.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:54.050+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:54.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:54.050+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:54.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:54.050+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:54.455 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:54.050+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:54.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:54.506+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:54.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:54.506+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:54.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:54.506+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:54.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:54.506+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:54.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:54.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:54.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:54.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:54.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:54.453+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:54.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:54.454+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:55.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:55.017+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:55.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:55.017+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:55.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:55.017+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:55.501 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:55.017+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:55.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:55.546+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:55.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:55.546+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:55.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:55.546+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:55.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:55.546+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:55.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:55.499+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:55.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:55.499+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:55.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:55.499+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:55.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:55.500+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:56.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:55.981+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:56.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:55.981+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:56.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:55.981+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:56.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:55.981+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:56.978 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:56.524+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:56.979 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:56.524+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:56.979 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:56.524+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:56.979 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:56.524+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:56.979 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:56.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:56.979 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:56.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:56.979 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:56.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:56.979 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:56.571+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:57.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:56.977+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:57.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:56.977+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:57.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:56.977+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:57.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:56.977+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:57.999 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:57.507+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:57.999 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:57.507+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:57.999 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:57.507+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:57.999 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:57.508+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:58.000 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:57.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:58.000 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:57.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:58.000 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:57.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:58.000 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:57.584+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:58.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:19:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:19:58.185+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 27099 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:19:58.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:57.998+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:58.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:57.998+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:58.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:57.998+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:58.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:57.998+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:58.548+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:58.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:58.548+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:58.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:58.548+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:58.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:58.548+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:58.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:58.460+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:58.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:58.460+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:58.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:58.460+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:58.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:58.461+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:59.478 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:59.036+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:59.478 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:59.036+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:59.478 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:59.036+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:19:59.478 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:19:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:19:59.036+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:59.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:59.535+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:59.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:59.535+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:59.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:59.535+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:19:59.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:19:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:19:59.535+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:19:59.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:59.477+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:59.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:59.477+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:59.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:59.477+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:19:59.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:19:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:19:59.477+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:00.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:00.060+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:00.060+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:00.060+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:00.060+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:01.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:00.510+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:00.510+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:00.510+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:00.510+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:01.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:00.568+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:01.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:00.568+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:01.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:00.568+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:01.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:00.568+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:01.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:01.043+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:01.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:01.043+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:01.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:01.043+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:01.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:01.043+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:02.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:01.534+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:02.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:01.534+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:02.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:01.534+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:02.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:01.534+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:02.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:01.616+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:02.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:01.616+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:02.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:01.616+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:02.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:01.616+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:02.034+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:02.034+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:02.034+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:02.034+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:03.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:02.553+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:03.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:02.553+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:03.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:02.553+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:03.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:02.554+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:03.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:02.661+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:03.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:02.661+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:03.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:02.661+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:03.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:02.661+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:03.315 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:20:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:20:03.189+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 27184 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:20:03.316 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:03.059+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:03.316 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:03.059+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:03.316 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:03.059+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:03.316 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:03.059+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:03.623 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:03.517+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:03.624 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:03.517+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:03.624 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:03.517+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:03.624 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:03.517+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:04.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:03.622+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:04.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:03.622+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:04.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:03.622+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:04.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:03.623+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:04.025+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:04.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:04.025+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:04.025+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:04.025+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:05.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:04.553+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:05.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:04.553+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:05.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:04.553+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:05.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:04.553+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:05.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:04.640+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:05.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:04.640+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:05.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:04.640+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:05.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:04.640+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:05.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:05.045+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:05.045+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:05.045+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:05.045+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:05.533 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:05.531+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:05.533 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:05.531+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:05.534 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:05.531+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:05.534 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:05.531+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:06.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:05.660+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:06.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:05.660+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:06.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:05.660+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:06.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:05.660+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:06.486 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:06.083+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:06.487 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:06.083+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:06.487 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:06.083+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:06.487 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:06.083+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:06.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:06.486+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:06.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:06.486+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:06.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:06.486+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:06.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:06.486+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:06.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:06.692+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:06.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:06.692+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:06.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:06.692+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:06.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:06.692+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:07.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:07.052+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:07.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:07.052+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:07.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:07.052+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:07.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:07.052+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:08.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:07.536+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:08.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:07.536+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:08.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:07.536+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:08.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:07.537+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:08.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:07.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:08.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:07.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:08.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:07.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:08.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:07.665+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:08.320 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:20:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:20:08.194+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 27268 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:20:08.320 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:08.013+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:08.321 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:08.013+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:08.321 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:08.013+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:08.321 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:08.013+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:09.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:08.571+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:09.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:08.571+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:09.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:08.571+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:09.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:08.571+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:09.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:08.636+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:09.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:08.636+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:09.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:08.636+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:09.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:08.636+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:09.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:09.016+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:09.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:09.016+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:09.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:09.016+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:09.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:09.016+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:09.968 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:09.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:09.968 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:09.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:09.968 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:09.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:09.968 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:09.587+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:09.969 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:09.600+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:09.969 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:09.600+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:09.969 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:09.600+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:09.969 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:09.600+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:10.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:09.967+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:10.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:09.967+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:10.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:09.967+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:10.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:09.967+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:10.992 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:10.548+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:10.993 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:10.548+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:10.993 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:10.548+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:10.993 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:10.548+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:10.993 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:10.619+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:10.993 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:10.619+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:10.994 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:10.619+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:10.994 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:10.619+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:11.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:10.991+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:11.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:10.991+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:11.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:10.991+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:11.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:10.991+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:11.965 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:11.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:11.965 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:11.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:11.965 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:11.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:11.965 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:11.587+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:11.965 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:11.578+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:11.966 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:11.578+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:11.966 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:11.578+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:11.966 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:11.578+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:12.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:11.963+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:12.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:11.963+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:12.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:11.963+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:12.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:11.964+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:12.975 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:12.541+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:12.976 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:12.541+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:12.976 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:12.541+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:12.976 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:12.542+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:12.976 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:12.561+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:12.976 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:12.561+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:12.976 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:12.561+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:12.977 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:12.561+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:13.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:20:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:20:13.198+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 27353 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:20:13.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:12.974+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:13.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:12.974+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:13.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:12.974+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:13.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:12.974+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:13.640 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:13.513+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:13.641 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:13.513+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:13.641 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:13.513+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:13.641 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:13.514+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:13.641 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:13.507+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:13.641 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:13.507+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:13.641 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:13.507+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:13.642 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:13.507+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:14.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:13.968+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:14.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:13.968+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:14.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:13.968+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:14.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:13.968+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:14.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:14.473+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:14.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:14.473+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:14.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:14.473+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:14.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:14.473+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:14.490+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:14.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:14.490+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:14.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:14.490+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:14.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:14.491+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:15.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:14.981+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:15.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:14.981+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:15.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:14.981+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:15.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:14.981+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:15.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:15.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:15.776 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:15.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:15.777 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:15.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:15.777 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:15.473+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:15.777 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:15.472+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:15.777 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:15.472+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:15.777 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:15.472+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:15.778 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:15.473+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:16.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:15.946+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:16.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:15.946+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:16.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:15.946+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:16.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:15.946+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:16.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:16.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:16.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:16.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:16.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:16.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:16.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:16.433+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:16.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:16.487+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:16.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:16.487+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:16.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:16.487+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:16.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:16.488+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:17.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:16.970+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:17.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:16.970+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:17.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:16.970+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:17.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:16.970+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:17.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:17.438+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:17.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:17.438+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:17.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:17.438+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:17.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:17.438+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:17.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:17.501+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:17.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:17.501+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:17.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:17.501+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:17.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:17.501+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:18.215 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:20:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:20:18.202+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 27436 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:20:18.215 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:17.963+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:18.216 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:17.963+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:18.216 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:17.963+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:18.216 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:17.963+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:18.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:18.514+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:18.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:18.514+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:18.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:18.514+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:18.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:18.514+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:18.468+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:18.468+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:18.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:18.468+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:18.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:18.468+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:19.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:18.960+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:19.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:18.960+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:19.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:18.960+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:19.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:18.960+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:19.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:19.496+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:19.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:19.496+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:19.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:19.496+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:19.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:19.496+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:19.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:19.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:19.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:19.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:19.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:19.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:19.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:19.430+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:20.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:19.954+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:20.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:19.954+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:20.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:19.954+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:20.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:19.954+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:20.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:20.443+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:20.443+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:20.443+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:20.443+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:20.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:20.462+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:20.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:20.462+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:20.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:20.462+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:20.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:20.462+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:21.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:20.905+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:21.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:20.905+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:21.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:20.905+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:21.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:20.905+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:21.744 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:21.455+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:21.745 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:21.455+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:21.745 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:21.455+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:21.745 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:21.455+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:21.745 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:21.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:21.745 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:21.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:21.745 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:21.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:21.746 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:21.404+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:22.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:21.950+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:22.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:21.950+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:22.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:21.950+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:22.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:21.951+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:22.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:22.376+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:22.376+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:22.376+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:22.376+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:22.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:22.478+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:22.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:22.478+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:22.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:22.478+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:22.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:22.478+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:23.208 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:22.917+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:23.208 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:22.917+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:23.209 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:22.917+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:23.209 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:22.917+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:23.497 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:20:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:20:23.207+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 27522 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:20:23.497 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:23.411+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:23.497 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:23.411+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:23.497 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:23.411+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:23.497 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:23.412+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:23.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:23.507+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:23.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:23.507+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:23.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:23.507+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:23.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:23.507+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:24.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:23.885+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:24.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:23.885+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:24.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:23.885+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:24.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:23.885+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:24.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:24.527+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:24.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:24.527+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:24.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:24.527+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:24.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:24.527+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:24.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:24.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:24.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:24.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:24.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:24.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:24.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:24.403+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:25.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:24.914+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:25.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:24.914+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:25.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:24.914+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:25.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:24.914+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:25.399+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:25.399+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:25.399+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:25.400+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:25.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:25.557+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:25.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:25.557+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:25.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:25.557+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:25.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:25.557+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:26.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:25.953+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:26.230 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:25.953+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:26.230 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:25.953+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:26.230 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:25.953+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:26.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:26.383+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:26.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:26.383+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:26.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:26.383+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:26.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:26.383+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:26.918 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:26.554+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:26.919 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:26.554+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:26.919 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:26.554+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:26.919 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:26.554+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:27.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:26.918+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:27.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:26.918+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:27.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:26.918+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:27.254 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:26.918+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:27.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:27.595+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:27.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:27.595+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:27.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:27.595+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:27.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:27.595+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:27.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:27.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:27.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:27.375+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:28.213 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:27.942+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:28.213 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:27.942+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:28.213 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:27.942+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:28.213 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:27.942+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:28.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:20:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:20:28.211+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 27605 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:20:28.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:28.345+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:28.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:28.345+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:28.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:28.345+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:28.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:28.345+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:28.976 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:28.560+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:28.976 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:28.560+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:28.977 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:28.560+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:28.977 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:28.560+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:29.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:28.975+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:29.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:28.975+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:29.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:28.975+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:29.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:28.975+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:29.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:29.549+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:29.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:29.549+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:29.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:29.549+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:29.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:29.549+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:29.365+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:29.365+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:29.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:29.365+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:29.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:29.366+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:30.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:29.933+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:30.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:29.933+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:30.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:29.933+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:30.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:29.933+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:30.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:30.552+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:30.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:30.552+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:30.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:30.552+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:30.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:30.552+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:30.360+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:30.360+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:30.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:30.360+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:30.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:30.361+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:31.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:30.976+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:31.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:30.976+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:31.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:30.976+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:31.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:30.976+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:31.745 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:31.524+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:31.746 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:31.524+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:31.746 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:31.524+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:31.746 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:31.524+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:31.746 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:31.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:31.746 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:31.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:31.747 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:31.404+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:31.747 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:31.404+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:32.372 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:32.010+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:32.372 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:32.010+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:32.373 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:32.010+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:32.373 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:32.010+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:32.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:32.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:32.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:32.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:32.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:32.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:32.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:32.426+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:32.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:32.477+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:32.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:32.477+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:32.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:32.477+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:32.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:32.477+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:33.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:20:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:20:33.216+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 27691 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:20:33.346 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:33.010+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:33.346 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:33.010+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:33.346 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:33.010+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:33.346 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:33.010+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:33.642 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:33.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:33.642 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:33.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:33.642 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:33.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:33.642 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:33.425+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:33.642 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:33.489+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:33.642 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:33.489+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:33.643 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:33.489+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:33.643 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:33.489+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:34.437 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:34.021+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:34.437 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:34.021+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:34.437 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:34.021+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:34.437 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:34.021+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:34.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:34.435+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:34.435+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:34.435+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:34.435+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:34.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:34.487+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:34.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:34.487+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:34.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:34.487+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:34.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:34.487+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:35.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:34.996+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:35.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:34.996+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:35.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:34.996+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:35.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:34.996+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:35.752 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:35.496+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:35.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:35.496+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:35.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:35.496+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:35.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:35.496+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:35.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:35.410+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:35.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:35.410+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:35.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:35.410+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:35.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:35.410+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:36.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:35.976+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:36.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:35.976+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:36.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:35.976+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:36.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:35.976+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:36.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:36.546+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:36.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:36.546+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:36.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:36.546+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:36.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:36.546+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:36.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:36.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:36.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:36.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:36.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:36.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:36.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:36.412+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:37.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:36.987+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:37.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:36.987+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:37.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:36.987+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:37.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:36.987+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:37.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:37.591+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:37.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:37.592+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:37.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:37.592+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:37.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:37.592+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:37.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:37.364+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:37.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:37.364+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:37.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:37.364+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:37.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:37.364+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:38.248 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:20:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:20:38.219+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 27774 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:20:38.249 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:37.987+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:38.249 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:37.987+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:38.249 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:37.987+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:38.249 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:37.987+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:38.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:38.347+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:38.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:38.347+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:38.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:38.347+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:38.506 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:38.348+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:38.967 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:38.596+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:38.968 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:38.596+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:38.968 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:38.596+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:38.968 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:38.596+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:39.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:38.965+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:39.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:38.965+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:39.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:38.965+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:39.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:38.965+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:39.627 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:39.303+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:39.627 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:39.303+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:39.627 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:39.303+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:39.627 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:39.303+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:40.000 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:39.626+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:40.000 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:39.626+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:40.000 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:39.626+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:40.000 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:39.626+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:40.252 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:39.998+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:40.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:39.998+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:40.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:39.998+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:40.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:39.998+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:40.642 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:40.272+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:40.643 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:40.272+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:40.643 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:40.272+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:40.643 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:40.272+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:41.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:40.640+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:41.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:40.640+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:41.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:40.640+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:41.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:40.640+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:41.310 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:41.019+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:41.311 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:41.019+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:41.311 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:41.019+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:41.311 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:41.019+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:41.593 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:41.309+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:41.593 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:41.309+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:41.593 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:41.309+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:41.593 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:41.309+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:42.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:41.591+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:42.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:41.591+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:42.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:41.591+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:42.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:41.591+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:42.292 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:42.012+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:42.292 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:42.012+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:42.292 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:42.012+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:42.292 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:42.012+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:42.631 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:42.291+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:42.632 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:42.291+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:42.632 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:42.291+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:42.632 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:42.291+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:43.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:42.629+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:43.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:42.629+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:43.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:42.629+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:43.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:42.629+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:43.341 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:20:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:20:43.224+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 27860 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:20:43.341 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:43.030+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:43.341 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:43.030+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:43.341 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:43.030+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:43.341 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:43.030+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:43.642 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:43.584+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:43.642 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:43.584+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:43.643 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:43.584+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:43.643 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:43.584+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:43.643 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:43.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:43.643 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:43.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:43.643 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:43.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:43.643 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:43.339+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:44.306 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:44.004+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:44.306 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:44.004+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:44.306 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:44.004+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:44.307 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:44.005+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:44.611 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:44.305+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:44.611 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:44.305+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:44.611 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:44.305+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:44.612 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:44.305+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:45.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:44.610+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:44.610+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:44.610+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:44.610+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:45.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:45.269+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:45.269+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:45.269+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:45.269+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:45.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:45.048+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:45.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:45.048+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:45.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:45.048+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:45.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:45.048+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:46.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:45.656+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:46.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:45.656+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:46.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:45.656+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:46.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:45.656+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:46.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:46.263+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:46.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:46.263+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:46.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:46.263+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:46.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:46.263+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:46.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:46.051+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:46.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:46.051+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:46.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:46.051+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:46.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:46.051+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:47.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:46.702+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:47.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:46.702+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:47.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:46.702+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:47.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:46.702+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:47.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:47.278+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:47.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:47.278+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:47.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:47.278+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:47.506 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:47.278+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:47.506 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:47.100+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:47.506 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:47.100+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:47.506 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:47.100+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:47.506 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:47.100+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:47.726+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:47.726+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:47.726+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:47.726+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:48.355 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:20:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:20:48.228+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 27943 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:20:48.355 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:48.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:48.355 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:48.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:48.356 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:48.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:48.356 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:48.098+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:48.356 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:48.251+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:48.356 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:48.251+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:48.356 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:48.251+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:48.356 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:48.251+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:49.054 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:48.761+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:49.054 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:48.761+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:49.054 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:48.761+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:49.055 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:48.761+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:49.267+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:49.267+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:49.267+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:49.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:49.267+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:49.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:49.053+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:49.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:49.053+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:49.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:49.053+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:49.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:49.053+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:50.099 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:49.775+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:50.100 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:49.775+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:50.100 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:49.775+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:50.100 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:49.775+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:50.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:50.265+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:50.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:50.265+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:50.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:50.265+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:50.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:50.265+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:50.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:50.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:50.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:50.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:50.098+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:50.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:50.098+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:51.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:50.733+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:51.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:50.733+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:51.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:50.733+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:51.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:50.733+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:51.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:51.229+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:51.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:51.229+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:51.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:51.229+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:51.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:51.229+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:51.059+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:51.059+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:51.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:51.059+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:51.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:51.059+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:52.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:51.713+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:51.713+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:51.713+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:51.714+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:52.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:52.233+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:52.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:52.233+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:52.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:52.233+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:52.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:52.233+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:52.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:52.035+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:52.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:52.035+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:52.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:52.035+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:52.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:52.035+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:53.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:52.699+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:53.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:52.699+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:53.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:52.699+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:53.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:52.699+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:53.358 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:20:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:20:53.233+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 28028 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:20:53.359 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:53.186+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:53.359 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:53.186+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:53.359 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:53.186+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:53.359 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:53.187+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:53.359 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:53.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:53.359 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:53.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:53.360 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:53.067+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:53.360 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:53.067+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:54.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:53.663+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:54.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:53.663+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:54.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:53.663+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:54.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:53.664+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:54.209+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:54.209+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:54.209+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:54.209+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:54.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:54.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:54.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:54.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:54.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:54.112+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:54.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:54.112+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:55.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:54.653+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:55.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:54.653+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:55.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:54.653+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:55.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:54.653+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:55.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:55.258+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:55.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:55.258+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:55.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:55.258+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:55.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:55.259+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:55.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:55.143+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:55.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:55.143+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:55.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:55.143+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:55.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:55.144+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:55.670+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:55.670+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:55.670+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:55.670+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:56.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:56.279+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:56.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:56.279+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:56.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:56.279+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:56.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:56.279+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:56.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:56.125+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:56.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:56.125+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:56.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:56.125+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:56.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:56.125+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:56.673+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:56.673+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:56.673+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:56.674+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:57.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:57.231+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:57.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:57.231+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:57.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:57.231+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:57.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:57.231+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:57.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:57.116+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:57.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:57.116+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:57.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:57.116+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:57.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:57.116+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:58.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:57.677+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:58.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:57.677+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:58.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:57.677+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:58.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:57.677+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:58.264 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:20:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:20:58.238+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 28111 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:20:58.264 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:58.214+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:58.264 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:58.214+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:58.265 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:58.214+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:58.265 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:58.214+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:58.265 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:58.123+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:58.265 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:58.123+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:58.265 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:58.123+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:58.265 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:58.124+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:59.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:58.727+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:59.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:58.727+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:59.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:58.727+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:20:59.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:58.728+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:59.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:59.232+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:59.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:59.232+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:59.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:59.232+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:20:59.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:20:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:20:59.232+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:20:59.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:59.127+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:59.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:59.127+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:59.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:59.127+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:20:59.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:20:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:20:59.127+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:00.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:59.728+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:00.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:59.728+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:00.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:59.728+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:00.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:20:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:20:59.729+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:00.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:00.206+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:00.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:00.206+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:00.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:00.206+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:00.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:00.207+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:00.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:00.142+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:00.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:00.142+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:00.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:00.142+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:00.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:00.142+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:01.116 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:00.771+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:01.116 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:00.771+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:01.116 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:00.771+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:01.117 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:00.771+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:01.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:01.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:01.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:01.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:01.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:01.115+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:01.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:01.115+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:01.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:01.191+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:01.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:01.191+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:01.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:01.191+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:01.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:01.191+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:02.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:01.739+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:02.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:01.739+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:02.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:01.739+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:02.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:01.740+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:02.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:02.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:02.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:02.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:02.113+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:02.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:02.178+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:02.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:02.178+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:02.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:02.178+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:02.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:02.178+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:03.093 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:02.786+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:03.093 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:02.786+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:03.093 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:02.786+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:03.093 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:02.787+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:03.368 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:21:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:21:03.242+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 28196 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:21:03.369 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:03.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:03.369 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:03.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:03.369 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:03.131+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:03.369 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:03.131+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:03.369 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:03.170+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:03.369 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:03.171+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:03.369 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:03.171+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:03.370 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:03.171+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:04.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:03.818+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:04.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:03.818+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:04.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:03.818+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:04.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:03.818+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:04.135+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:04.135+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:04.135+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:04.136+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:04.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:04.141+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:04.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:04.141+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:04.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:04.141+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:04.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:04.141+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:05.114 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:04.840+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:05.114 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:04.840+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:05.115 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:04.840+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:05.115 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:04.841+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:05.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:05.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:05.113+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:05.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:05.113+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:05.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:05.137+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:05.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:05.137+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:05.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:05.137+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:05.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:05.138+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:06.160 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:05.820+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:06.160 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:05.820+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:06.160 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:05.820+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:06.160 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:05.820+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:06.502 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:06.162+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:06.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:06.162+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:06.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:06.162+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:06.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:06.162+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:06.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:06.158+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:06.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:06.158+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:06.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:06.158+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:06.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:06.159+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:07.113 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:06.852+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:07.113 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:06.852+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:07.113 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:06.852+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:07.114 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:06.853+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:07.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:07.182+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:07.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:07.182+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:07.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:07.182+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:07.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:07.182+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:07.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:07.111+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:07.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:07.111+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:07.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:07.111+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:07.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:07.111+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:08.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:07.890+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:08.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:07.890+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:08.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:07.890+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:08.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:07.890+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:08.214 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:08.084+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:08.214 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:08.084+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:08.214 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:08.084+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:08.214 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:08.085+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:08.502 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:21:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:21:08.246+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 28279 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:21:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:08.211+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:08.211+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:08.211+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:08.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:08.212+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:09.219 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:08.915+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:09.219 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:08.915+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:09.219 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:08.915+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:09.220 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:08.916+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:09.220 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:09.055+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:09.220 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:09.055+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:09.220 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:09.055+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:09.220 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:09.055+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:09.488 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:09.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:09.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:09.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:09.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:09.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:09.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:09.247+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:10.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:09.945+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:10.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:09.945+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:10.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:09.945+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:10.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:09.946+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:10.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:10.022+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:10.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:10.022+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:10.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:10.022+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:10.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:10.022+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:10.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:10.252+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:10.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:10.252+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:10.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:10.252+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:10.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:10.252+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:11.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:10.912+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:11.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:10.912+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:11.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:10.912+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:11.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:10.913+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:11.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:11.063+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:11.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:11.063+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:11.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:11.063+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:11.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:11.064+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:11.748 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:11.280+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:11.749 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:11.280+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:11.749 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:11.280+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:11.749 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:11.280+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:11.881+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:11.881+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:11.881+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:11.882+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:12.308 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:12.051+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:12.308 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:12.051+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:12.308 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:12.051+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:12.309 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:12.051+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:12.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:12.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:12.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:12.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:12.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:12.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:12.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:12.307+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:13.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:13.007+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:13.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:13.007+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:13.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:13.007+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:13.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:13.008+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:13.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:12.858+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:13.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:12.858+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:13.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:12.858+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:13.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:12.858+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:13.644 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:21:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:21:13.251+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 28365 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:21:13.644 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:13.352+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:13.644 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:13.352+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:13.645 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:13.352+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:13.645 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:13.352+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:14.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:13.813+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:14.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:13.813+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:14.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:13.813+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:14.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:13.814+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:14.340 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:14.038+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:14.340 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:14.038+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:14.340 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:14.038+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:14.340 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:14.038+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:14.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:14.338+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:14.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:14.338+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:14.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:14.338+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:14.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:14.338+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:15.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:14.997+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:15.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:14.997+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:15.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:14.997+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:15.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:14.997+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:15.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:14.767+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:15.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:14.767+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:15.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:14.767+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:15.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:14.768+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:15.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:15.334+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:15.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:15.334+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:15.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:15.334+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:15.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:15.334+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:16.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:15.970+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:16.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:15.970+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:16.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:15.970+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:16.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:15.970+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:16.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:15.790+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:16.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:15.790+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:16.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:15.790+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:16.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:15.790+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:16.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:16.329+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:16.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:16.329+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:16.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:16.329+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:16.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:16.329+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:17.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:16.977+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:17.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:16.977+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:17.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:16.977+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:17.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:16.978+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:17.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:16.808+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:17.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:16.808+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:17.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:16.808+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:17.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:16.809+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:17.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:17.377+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:17.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:17.377+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:17.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:17.377+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:17.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:17.377+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:18.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:17.984+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:18.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:17.984+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:18.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:17.984+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:18.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:17.985+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:18.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:17.769+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:18.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:17.769+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:18.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:17.769+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:18.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:17.769+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:18.631 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:21:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:21:18.256+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 28448 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:21:18.631 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:18.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:18.631 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:18.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:18.631 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:18.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:18.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:18.390+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:18.949 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:18.781+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:18.950 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:18.781+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:18.950 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:18.781+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:18.950 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:18.782+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:19.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:18.948+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:19.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:18.948+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:19.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:18.948+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:19.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:18.948+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:19.436+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:19.436+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:19.436+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:19.437+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:20.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:19.970+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:20.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:19.970+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:20.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:19.970+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:20.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:19.971+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:20.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:19.812+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:20.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:19.812+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:20.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:19.812+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:20.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:19.812+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:20.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:20.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:20.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:20.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:20.396+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:20.778+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:20.778+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:20.778+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:20.779+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:21.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:20.957+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:21.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:20.957+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:21.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:20.957+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:21.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:20.957+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:21.749 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:21.392+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:21.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:21.393+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:21.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:21.393+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:21.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:21.393+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:22.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:21.932+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:21.932+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:21.932+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:22.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:21.933+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:22.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:21.811+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:22.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:21.811+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:22.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:21.811+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:22.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:21.811+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:22.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:22.407+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:22.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:22.407+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:22.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:22.407+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:22.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:22.407+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:23.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:22.785+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:23.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:22.785+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:23.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:22.785+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:23.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:22.785+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:23.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:22.925+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:23.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:22.925+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:23.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:22.925+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:23.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:22.925+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:23.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:21:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:21:23.260+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 28534 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:21:23.645 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:23.412+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:23.645 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:23.412+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:23.645 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:23.412+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:23.646 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:23.412+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:24.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:23.788+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:24.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:23.788+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:24.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:23.788+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:24.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:23.788+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:24.006 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:23.886+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:24.006 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:23.886+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:24.006 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:23.886+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:24.006 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:23.886+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:24.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:24.406+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:24.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:24.406+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:24.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:24.406+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:24.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:24.406+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:25.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:24.892+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:25.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:24.892+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:25.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:24.892+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:25.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:24.892+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:25.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:24.762+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:25.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:24.762+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:25.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:24.762+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:25.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:24.763+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:25.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:25.420+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:25.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:25.420+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:25.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:25.420+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:25.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:25.421+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:26.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:25.904+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:26.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:25.904+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:26.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:25.904+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:26.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:25.904+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:26.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:25.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:26.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:25.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:26.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:25.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:26.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:25.801+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:26.427+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:26.427+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:26.427+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:26.428+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:27.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:26.883+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:27.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:26.883+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:27.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:26.883+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:27.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:26.883+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:27.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:26.774+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:27.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:26.774+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:27.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:26.774+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:27.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:26.775+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:27.742 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:27.394+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:27.742 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:27.394+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:27.742 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:27.394+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:27.743 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:27.394+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:28.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:27.741+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:28.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:27.741+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:28.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:27.741+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:28.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:27.742+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:28.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:27.845+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:28.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:27.845+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:28.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:27.845+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:28.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:27.845+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:28.752 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:21:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:21:28.264+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 28617 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:21:28.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:28.386+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:28.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:28.386+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:28.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:28.386+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:28.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:28.386+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:29.196 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:28.882+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:29.196 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:28.882+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:29.197 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:28.882+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:29.197 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:28.883+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:29.197 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:28.774+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:29.197 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:28.774+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:29.197 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:28.774+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:29.197 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:28.774+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:29.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:29.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:29.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:29.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:29.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:29.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:29.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:29.370+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:30.199 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:29.784+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:30.199 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:29.784+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:30.200 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:29.784+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:30.200 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:29.785+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:30.200 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:29.852+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:30.200 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:29.852+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:30.200 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:29.852+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:30.200 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:29.852+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:30.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:30.358+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:30.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:30.358+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:30.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:30.358+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:30.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:30.358+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:31.002 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:30.832+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:31.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:30.832+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:31.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:30.832+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:31.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:30.832+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:31.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:30.737+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:31.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:30.737+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:31.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:30.737+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:31.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:30.738+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:31.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:31.370+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:31.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:31.370+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:31.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:31.370+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:31.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:31.370+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:32.002 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:31.783+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:32.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:31.783+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:32.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:31.783+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:32.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:31.783+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:32.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:31.874+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:32.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:31.874+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:32.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:31.874+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:32.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:31.874+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:32.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:32.375+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:32.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:32.375+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:32.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:32.375+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:32.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:32.375+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:33.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:32.814+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:33.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:32.814+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:33.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:32.814+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:33.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:32.814+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:33.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:32.894+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:33.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:32.894+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:33.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:32.894+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:33.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:32.894+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:33.645 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:21:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:21:33.269+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 28703 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:21:33.646 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:33.400+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:33.646 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:33.400+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:33.646 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:33.400+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:33.646 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:33.401+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:33.903 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:33.770+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:33.903 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:33.770+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:33.903 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:33.770+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:33.903 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:33.771+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:34.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:33.901+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:34.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:33.901+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:34.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:33.901+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:34.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:33.902+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:34.747 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:34.434+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:34.747 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:34.434+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:34.748 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:34.434+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:34.748 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:34.435+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:35.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:34.746+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:35.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:34.746+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:35.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:34.746+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:35.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:34.746+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:35.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:34.924+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:35.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:34.924+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:35.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:34.924+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:35.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:34.924+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:35.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:35.480+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:35.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:35.480+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:35.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:35.480+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:35.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:35.480+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:36.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:35.753+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:36.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:35.753+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:36.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:35.753+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:36.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:35.754+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:36.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:35.903+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:36.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:35.903+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:36.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:35.903+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:36.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:35.903+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:36.868 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:36.719+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:36.868 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:36.719+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:36.868 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:36.719+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:36.869 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:36.719+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:36.869 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:36.511+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:36.869 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:36.511+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:36.869 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:36.511+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:36.869 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:36.511+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:37.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:36.867+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:37.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:36.867+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:37.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:36.867+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:37.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:36.867+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:37.826 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:37.686+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:37.826 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:37.686+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:37.826 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:37.686+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:37.826 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:37.686+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:37.826 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:37.543+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:37.827 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:37.543+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:37.827 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:37.543+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:37.827 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:37.543+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:38.252 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:37.825+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:38.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:37.825+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:38.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:37.825+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:38.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:37.825+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:38.578 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:21:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:21:38.274+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 28786 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:21:39.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:38.676+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:39.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:38.676+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:39.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:38.676+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:39.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:38.676+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:39.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:38.577+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:39.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:38.577+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:39.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:38.577+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:39.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:38.577+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:39.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:38.779+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:39.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:38.779+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:39.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:38.779+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:39.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:38.779+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:39.765+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:40.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:39.765+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:40.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:39.765+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:40.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:39.765+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:40.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:39.668+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:40.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:39.668+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:40.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:39.668+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:40.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:39.668+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:40.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:39.579+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:40.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:39.579+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:40.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:39.579+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:40.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:39.579+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:40.702 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:40.530+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:40.702 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:40.530+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:40.702 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:40.530+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:40.702 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:40.530+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:41.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:40.701+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:41.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:40.701+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:41.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:40.701+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:41.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:40.701+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:41.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:40.743+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:41.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:40.743+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:41.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:40.743+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:41.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:40.743+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:42.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:41.661+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:42.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:41.661+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:42.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:41.661+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:42.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:41.661+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:42.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:41.559+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:42.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:41.559+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:42.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:41.559+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:42.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:41.559+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:42.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:41.790+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:42.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:41.790+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:42.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:41.790+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:42.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:41.791+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:42.833 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:42.644+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:42.834 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:42.644+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:42.834 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:42.644+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:42.834 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:42.645+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:42.834 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:42.527+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:42.834 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:42.527+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:42.834 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:42.527+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:42.834 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:42.527+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:43.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:42.832+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:43.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:42.832+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:43.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:42.832+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:43.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:42.832+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:43.646 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:21:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:21:43.279+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 28872 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:21:43.647 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:43.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:43.647 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:43.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:43.647 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:43.508+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:43.647 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:43.508+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:44.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:43.668+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:44.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:43.668+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:44.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:43.668+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:44.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:43.668+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:44.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:43.842+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:44.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:43.842+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:44.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:43.842+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:44.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:43.842+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:44.646+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:44.646+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:44.646+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:44.646+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:45.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:44.793+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:45.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:44.793+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:45.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:44.793+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:45.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:44.793+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:45.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:44.551+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:45.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:44.552+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:45.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:44.552+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:45.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:44.552+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:46.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:45.634+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:46.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:45.634+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:46.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:45.634+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:46.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:45.634+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:46.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:45.779+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:46.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:45.779+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:46.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:45.779+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:46.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:45.780+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:46.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:45.596+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:46.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:45.596+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:46.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:45.596+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:46.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:45.597+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:47.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:46.741+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:47.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:46.741+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:47.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:46.741+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:47.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:46.741+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:47.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:46.680+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:47.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:46.680+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:47.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:46.680+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:47.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:46.680+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:47.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:46.576+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:47.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:46.576+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:47.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:46.576+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:47.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:46.576+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:47.646+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:47.646+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:47.646+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:47.647+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:48.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:47.702+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:48.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:47.702+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:48.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:47.702+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:48.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:47.702+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:48.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:47.609+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:48.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:47.610+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:48.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:47.610+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:48.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:47.610+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:48.616 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:21:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:21:48.283+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 28955 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:21:49.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:48.650+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:49.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:48.650+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:49.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:48.650+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:49.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:48.650+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:49.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:48.615+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:49.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:48.615+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:49.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:48.615+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:49.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:48.615+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:49.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:48.703+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:49.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:48.703+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:49.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:48.703+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:49.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:48.703+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:50.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:49.666+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:50.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:49.666+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:50.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:49.666+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:50.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:49.666+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:50.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:49.723+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:50.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:49.723+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:50.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:49.723+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:50.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:49.723+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:50.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:49.628+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:50.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:49.628+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:50.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:49.628+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:50.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:49.628+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:51.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:50.625+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:51.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:50.625+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:51.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:50.625+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:51.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:50.625+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:51.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:50.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:51.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:50.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:51.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:50.735+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:51.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:50.736+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:51.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:50.632+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:51.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:50.632+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:51.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:50.632+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:51.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:50.632+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:51.651+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:51.651+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:51.651+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:51.651+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:52.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:51.741+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:52.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:51.741+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:52.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:51.741+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:52.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:51.741+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:52.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:51.626+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:52.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:51.626+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:52.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:51.626+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:52.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:51.626+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:53.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:52.684+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:53.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:52.684+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:53.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:52.684+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:53.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:52.684+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:53.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:52.726+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:53.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:52.726+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:53.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:52.726+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:53.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:52.726+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:53.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:52.587+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:53.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:52.587+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:53.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:52.587+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:53.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:52.587+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:53.569 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:21:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:21:53.288+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 29041 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:21:54.002 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:53.567+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:54.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:53.567+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:54.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:53.567+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:54.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:53.568+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:54.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:53.762+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:54.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:53.762+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:54.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:53.762+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:54.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:53.763+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:54.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:53.650+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:54.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:53.650+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:54.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:53.650+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:54.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:53.651+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:55.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:54.653+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:55.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:54.653+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:55.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:54.653+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:55.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:54.653+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:55.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:54.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:55.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:54.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:55.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:54.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:55.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:54.777+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:55.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:54.585+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:55.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:54.585+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:55.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:54.585+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:55.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:54.585+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:55.629 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:55.554+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:55.629 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:55.554+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:55.629 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:55.554+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:55.629 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:55.554+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:55.626+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:55.626+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:55.626+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:55.627+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:56.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:55.791+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:56.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:55.791+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:56.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:55.791+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:56.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:55.792+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:56.584+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:56.584+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:56.584+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:56.584+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:57.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:56.748+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:57.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:56.748+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:57.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:56.748+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:57.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:56.748+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:57.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:56.537+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:57.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:56.537+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:57.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:56.537+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:57.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:56.537+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:57.766 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:57.614+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:57.766 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:57.614+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:57.766 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:57.614+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:57.766 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:57.614+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:57.766 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:57.502+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:57.767 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:57.502+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:57.767 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:57.502+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:57.767 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:57.502+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:58.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:57.764+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:58.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:57.764+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:58.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:57.764+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:58.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:57.764+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:58.577 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:21:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:21:58.292+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 29124 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:21:58.577 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:58.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:58.577 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:58.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:58.578 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:58.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:21:58.578 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:58.529+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:59.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:58.575+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:59.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:58.575+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:59.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:58.575+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:21:59.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:58.575+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:21:59.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:58.794+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:59.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:58.794+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:59.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:58.794+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:21:59.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:58.794+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:00.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:59.591+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:00.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:59.591+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:00.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:59.591+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:00.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:21:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:21:59.591+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:00.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:59.556+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:00.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:59.556+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:00.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:59.556+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:00.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:21:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:21:59.556+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:00.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:59.748+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:00.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:59.748+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:00.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:59.748+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:00.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:21:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:21:59.748+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:00.627+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:00.627+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:00.627+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:00.627+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:01.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:00.534+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:01.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:00.534+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:01.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:00.534+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:01.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:00.534+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:01.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:00.706+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:01.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:00.706+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:01.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:00.706+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:01.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:00.706+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:02.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:01.642+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:02.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:01.642+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:02.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:01.642+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:02.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:01.642+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:02.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:01.684+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:02.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:01.684+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:02.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:01.684+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:02.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:01.684+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:02.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:01.581+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:02.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:01.581+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:02.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:01.581+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:02.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:01.581+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:03.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:02.648+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:03.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:02.648+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:03.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:02.648+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:03.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:02.648+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:03.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:02.658+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:03.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:02.658+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:03.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:02.658+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:03.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:02.659+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:03.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:02.552+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:03.006 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:02.552+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:03.006 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:02.552+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:03.006 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:02.552+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:03.648 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:22:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:22:03.297+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 29209 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:22:03.649 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:03.530+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:03.649 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:03.530+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:03.649 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:03.530+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:03.649 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:03.530+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:04.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:03.677+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:04.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:03.677+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:04.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:03.677+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:04.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:03.678+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:04.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:03.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:04.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:03.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:04.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:03.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:04.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:03.662+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:04.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:04.653+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:04.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:04.653+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:04.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:04.653+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:04.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:04.653+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:04.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:04.672+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:04.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:04.672+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:04.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:04.672+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:04.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:04.672+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:04.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:04.499+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:04.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:04.499+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:04.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:04.499+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:04.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:04.499+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:06.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:05.709+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:06.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:05.709+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:06.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:05.709+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:06.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:05.709+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:06.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:05.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:06.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:05.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:06.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:05.630+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:06.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:05.630+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:06.006 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:05.515+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:06.006 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:05.515+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:06.006 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:05.515+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:06.006 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:05.515+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:07.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:06.623+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:07.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:06.623+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:07.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:06.623+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:07.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:06.623+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:07.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:06.670+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:07.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:06.670+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:07.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:06.670+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:07.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:06.671+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:07.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:06.565+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:07.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:06.565+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:07.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:06.565+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:07.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:06.565+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:08.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:07.647+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:08.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:07.647+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:08.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:07.647+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:08.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:07.647+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:08.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:07.592+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:08.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:07.592+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:08.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:07.592+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:08.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:07.592+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:08.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:07.600+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:08.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:07.600+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:08.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:07.600+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:08.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:07.600+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:08.565 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:22:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:22:08.301+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 29283 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:22:09.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:08.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:09.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:08.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:09.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:08.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:09.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:08.585+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:09.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:08.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:09.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:08.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:09.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:08.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:09.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:08.563+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:09.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:08.640+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:09.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:08.640+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:09.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:08.640+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:09.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:08.640+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:10.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:09.658+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:10.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:09.658+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:10.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:09.658+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:10.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:09.658+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:10.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:09.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:10.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:09.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:10.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:09.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:10.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:09.556+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:10.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:09.607+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:10.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:09.607+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:10.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:09.607+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:10.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:09.607+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:11.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:10.548+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:11.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:10.548+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:11.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:10.548+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:11.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:10.548+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:11.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:10.697+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:11.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:10.697+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:11.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:10.697+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:11.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:10.697+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:11.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:10.634+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:11.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:10.634+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:11.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:10.634+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:11.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:10.634+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:11.674+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:11.674+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:11.674+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:11.675+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:12.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:11.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:12.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:11.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:12.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:11.512+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:12.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:11.512+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:12.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:11.661+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:12.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:11.661+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:12.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:11.661+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:12.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:11.661+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:13.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:12.532+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:13.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:12.532+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:13.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:12.532+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:13.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:12.532+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:13.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:12.664+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:13.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:12.664+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:13.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:12.664+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:13.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:12.664+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:13.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:12.650+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:13.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:12.650+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:13.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:12.650+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:13.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:12.651+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:13.618 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:22:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:22:13.307+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 29368 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:22:13.619 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:13.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:13.619 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:13.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:13.619 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:13.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:13.619 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:13.551+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:14.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:13.636+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:14.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:13.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:14.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:13.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:14.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:13.637+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:14.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:13.617+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:14.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:13.617+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:14.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:13.617+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:14.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:13.617+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:15.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:14.516+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:15.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:14.516+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:15.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:14.516+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:15.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:14.517+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:15.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:14.596+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:15.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:14.596+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:15.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:14.596+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:15.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:14.597+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:15.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:14.581+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:15.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:14.581+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:15.006 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:14.581+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:15.006 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:14.582+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:15.600+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:15.600+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:15.600+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:15.601+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:16.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:15.547+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:16.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:15.547+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:16.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:15.547+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:16.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:15.548+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:16.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:15.601+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:16.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:15.601+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:16.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:15.601+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:16.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:15.601+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:17.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:16.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:17.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:16.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:17.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:16.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:17.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:16.577+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:17.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:16.552+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:17.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:16.552+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:17.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:16.552+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:17.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:16.553+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:17.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:16.633+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:17.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:16.633+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:17.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:16.633+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:17.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:16.634+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:18.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:17.523+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:18.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:17.523+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:18.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:17.523+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:18.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:17.524+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:18.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:17.623+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:18.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:17.623+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:18.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:17.623+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:18.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:17.623+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:18.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:17.589+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:18.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:17.589+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:18.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:17.589+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:18.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:17.589+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:18.572 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:22:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:22:18.311+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 29451 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:22:19.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:18.633+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:19.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:18.633+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:19.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:18.633+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:19.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:18.633+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:19.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:18.589+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:19.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:18.589+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:19.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:18.589+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:19.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:18.590+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:19.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:18.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:19.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:18.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:19.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:18.571+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:19.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:18.571+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:20.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:19.599+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:20.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:19.599+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:20.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:19.599+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:20.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:19.600+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:20.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:19.631+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:20.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:19.631+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:20.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:19.631+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:20.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:19.632+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:20.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:19.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:20.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:19.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:20.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:19.612+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:20.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:19.612+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:21.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:20.599+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:21.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:20.599+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:21.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:20.599+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:21.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:20.600+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:21.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:20.675+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:21.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:20.675+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:21.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:20.675+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:21.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:20.676+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:21.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:20.570+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:21.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:20.570+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:21.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:20.570+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:21.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:20.571+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:22.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:21.567+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:22.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:21.567+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:22.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:21.567+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:22.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:21.568+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:22.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:21.683+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:22.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:21.683+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:22.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:21.683+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:22.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:21.684+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:22.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:21.531+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:22.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:21.531+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:22.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:21.531+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:22.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:21.531+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:22.964 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:22.541+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:22.965 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:22.541+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:22.965 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:22.541+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:22.965 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:22.541+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:22.965 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:22.727+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:22.966 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:22.727+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:22.966 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:22.727+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:22.966 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:22.728+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:22.966 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:22.578+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:22.966 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:22.578+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:22.966 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:22.578+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:22.966 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:22.579+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:23.590 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:22:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:22:23.316+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 29537 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:22:23.591 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:23.506+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:23.591 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:23.506+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:23.591 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:23.506+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:23.591 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:23.507+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:24.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:23.750+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:24.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:23.750+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:24.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:23.750+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:24.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:23.751+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:24.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:23.589+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:24.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:23.589+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:24.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:23.589+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:24.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:23.589+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:24.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:24.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:24.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:24.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:24.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:24.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:24.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:24.556+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:24.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:24.482+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:24.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:24.482+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:24.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:24.482+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:24.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:24.483+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:24.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:24.727+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:24.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:24.727+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:24.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:24.727+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:24.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:24.728+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:25.746 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:25.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:25.746 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:25.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:25.746 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:25.585+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:25.746 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:25.585+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:25.747 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:25.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:25.747 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:25.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:25.747 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:25.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:25.747 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:25.479+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:26.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:25.744+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:26.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:25.744+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:26.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:25.744+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:26.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:25.745+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:26.461+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:26.461+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:26.461+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:26.461+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:26.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:26.708+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:26.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:26.708+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:26.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:26.708+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:26.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:26.708+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:26.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:26.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:26.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:26.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:26.610+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:28.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:27.504+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:28.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:27.504+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:28.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:27.504+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:28.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:27.505+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:28.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:27.684+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:28.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:27.684+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:28.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:27.684+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:28.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:27.684+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:28.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:27.573+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:28.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:27.573+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:28.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:27.573+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:28.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:27.574+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:28.588 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:22:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:22:28.321+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 29620 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:22:28.588 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:28.552+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:28.588 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:28.552+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:28.589 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:28.552+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:28.589 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:28.552+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:29.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:28.660+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:29.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:28.660+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:29.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:28.660+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:29.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:28.661+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:29.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:28.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:29.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:28.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:29.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:28.586+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:29.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:28.587+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:30.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:29.547+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:30.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:29.547+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:30.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:29.547+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:30.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:29.547+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:30.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:29.618+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:30.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:29.618+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:30.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:29.618+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:30.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:29.619+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:30.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:29.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:30.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:29.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:30.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:29.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:30.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:29.529+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:30.501+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:30.501+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:30.501+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:30.502+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:30.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:30.662+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:30.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:30.662+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:30.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:30.662+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:30.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:30.662+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:30.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:30.483+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:30.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:30.483+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:30.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:30.483+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:30.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:30.483+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:31.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:31.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:31.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:31.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:31.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:31.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:31.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:31.526+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:31.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:31.670+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:31.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:31.670+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:31.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:31.670+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:31.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:31.671+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:31.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:31.500+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:31.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:31.500+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:31.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:31.500+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:31.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:31.500+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:32.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:32.704+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:32.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:32.704+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:32.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:32.704+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:32.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:32.704+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:32.545+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:32.545+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:32.545+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:32.546+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:32.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:32.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:32.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:32.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:32.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:32.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:32.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:32.474+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:33.651 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:22:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:22:33.325+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 29706 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:22:33.652 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:33.471+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:33.652 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:33.471+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:33.652 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:33.471+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:33.652 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:33.472+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:33.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:33.563+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:33.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:33.563+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:33.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:33.563+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:33.653 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:33.564+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:34.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:33.753+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:34.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:33.753+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:34.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:33.753+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:05.984356+0000 (oldest deadline 2023-12-16T21:12:25.984356+0000) 2023-12-16T21:22:34.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:33.754+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:34.707 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:34.609+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:34.707 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:34.609+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:34.707 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:34.609+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:34.707 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:34.610+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:34.708 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:34.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:34.708 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:34.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:34.708 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:34.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:11:59.032580+0000 (oldest deadline 2023-12-16T21:12:19.032580+0000) 2023-12-16T21:22:34.708 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:34.449+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:35.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:34.705+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:35.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:35.499+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:36.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:35.731+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:36.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:35.652+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:36.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:35.652+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:36.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:35.652+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:36.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:35.652+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:36.737 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:36.479+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:36.738 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:36.607+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:36.738 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:36.607+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:36.739 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:36.607+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:36.739 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:36.607+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:37.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:36.735+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:38.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:37.691+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:38.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:37.524+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:38.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:37.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:38.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:37.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:38.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:37.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:38.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:37.634+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:38.600 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:22:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:22:38.330+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 29789 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:22:38.600 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:38.532+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:39.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:38.725+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:39.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:38.598+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:39.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:38.598+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:39.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:38.598+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:39.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:38.599+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:39.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:39.485+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:39.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:39.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:39.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:39.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:39.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:39.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:39.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:39.633+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:40.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:39.752+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:40.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:40.457+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:40.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:40.597+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:40.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:40.597+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:40.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:40.597+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:40.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:40.598+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:41.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:40.777+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:41.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:41.441+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:41.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:41.627+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:41.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:41.627+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:41.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:41.627+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:41.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:41.627+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:42.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:41.777+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:42.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:42.463+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:42.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:42.619+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:42.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:42.619+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:42.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:42.619+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:12:08.783288+0000 (oldest deadline 2023-12-16T21:12:28.783288+0000) 2023-12-16T21:22:42.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:42.619+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:43.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:42.758+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:43.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:22:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:22:43.335+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 29875 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:22:43.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:43.573+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:43.653 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:43.484+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:44.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:43.804+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:44.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:44.553+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:44.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:44.519+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:45.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:44.847+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:45.850 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:45.573+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:45.851 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:45.526+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:46.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:45.848+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:46.837 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:46.575+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:46.837 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:46.507+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:47.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:46.836+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:47.860 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:47.539+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:47.860 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:47.503+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:48.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:47.859+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:48.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:22:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:22:48.340+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 29958 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:22:48.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:48.569+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:48.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:48.454+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:49.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:48.836+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:49.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:49.578+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:49.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:49.413+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:50.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:49.833+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:50.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:50.549+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:50.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:50.397+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:51.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:50.807+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:51.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:51.537+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:51.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:51.436+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:52.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:51.820+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:52.498+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:52.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:52.476+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:53.252 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:52.844+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:53.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:22:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:22:53.345+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 30043 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:22:53.654 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:53.506+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:53.654 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:53.492+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:54.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:53.831+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:54.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:54.492+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:54.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:54.504+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:55.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:54.797+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:55.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:55.463+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:55.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:55.511+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:55.838+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:22:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:55.838+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:22:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:55.838+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:22:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:55.838+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:56.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:56.499+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:56.462+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:57.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:56.836+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:22:57.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:56.836+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:22:57.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:56.836+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:22:57.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:56.837+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:57.789 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:57.508+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:57.789 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:57.507+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:58.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:57.788+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:22:58.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:57.788+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:22:58.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:57.788+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:22:58.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:57.788+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:58.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:58.480+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:58.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:58.508+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:58.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:22:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:22:58.349+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 30125 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:22:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:58.825+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:22:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:58.825+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:22:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:58.825+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:22:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:58.825+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:59.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:22:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:22:59.492+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:22:59.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:22:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:22:59.495+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:00.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:59.787+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:00.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:59.787+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:00.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:59.787+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:00.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:22:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:22:59.787+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:00.742 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:00.475+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:00.742 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:00.462+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:00.743 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:00.462+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:00.743 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:00.462+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:00.743 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:00.462+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:00.741+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:00.741+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:00.741+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:00.741+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:01.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:01.433+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:01.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:01.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:01.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:01.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:01.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:01.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:01.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:01.454+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:02.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:01.767+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:02.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:01.767+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:02.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:01.767+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:02.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:01.768+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:02.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:02.466+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:02.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:02.481+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:02.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:02.481+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:02.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:02.481+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:02.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:02.482+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:03.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:02.789+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:03.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:02.789+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:03.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:02.789+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:03.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:02.789+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:03.653 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:23:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:23:03.354+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 30211 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:23:03.654 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:03.468+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:03.654 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:03.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:03.654 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:03.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:03.654 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:03.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:03.654 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:03.454+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:04.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:03.811+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:04.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:03.811+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:04.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:03.811+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:04.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:03.811+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:04.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:04.467+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:04.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:04.467+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:04.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:04.467+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:04.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:04.468+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:04.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:04.464+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:04.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:04.464+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:04.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:04.464+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:04.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:04.464+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:05.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:04.838+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:05.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:04.838+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:05.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:04.838+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:05.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:04.838+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:05.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:05.486+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:05.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:05.486+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:05.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:05.486+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:05.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:05.486+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:05.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:05.460+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:05.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:05.460+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:05.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:05.460+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:05.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:05.460+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:06.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:05.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:06.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:05.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:06.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:05.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:06.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:05.802+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:06.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:06.461+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:06.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:06.461+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:06.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:06.461+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:06.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:06.462+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:06.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:06.461+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:06.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:06.461+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:06.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:06.461+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:06.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:06.461+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:06.756+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:06.756+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:06.756+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:06.757+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:07.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:07.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:07.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:07.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:07.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:07.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:07.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:07.425+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:07.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:07.464+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:07.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:07.464+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:07.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:07.464+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:07.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:07.464+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:08.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:07.778+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:08.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:07.778+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:08.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:07.778+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:08.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:07.778+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:08.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:23:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:23:08.359+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 30290 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:23:08.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:08.388+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:08.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:08.388+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:08.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:08.388+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:08.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:08.388+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:08.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:08.421+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:08.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:08.421+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:08.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:08.421+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:08.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:08.421+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:08.796+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:08.796+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:08.796+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:08.796+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:09.366+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:09.366+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:09.366+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:09.367+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:09.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:09.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:09.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:09.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:09.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:09.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:09.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:09.466+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:10.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:09.803+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:10.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:09.803+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:10.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:09.803+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:10.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:09.803+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:10.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:10.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:10.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:10.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:10.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:10.417+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:10.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:10.417+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:10.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:10.499+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:10.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:10.499+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:10.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:10.499+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:10.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:10.500+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:11.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:10.819+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:11.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:10.819+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:11.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:10.819+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:11.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:10.820+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:11.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:11.384+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:11.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:11.384+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:11.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:11.384+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:11.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:11.384+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:11.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:11.488+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:11.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:11.488+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:11.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:11.488+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:11.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:11.488+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:12.117 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:11.777+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:12.117 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:11.777+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:12.117 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:11.777+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:12.117 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:11.777+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:12.449 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:12.352+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:12.449 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:12.352+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:12.449 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:12.352+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:12.449 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:12.352+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:12.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:12.446+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:12.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:12.446+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:12.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:12.446+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:12.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:12.446+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:13.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:12.819+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:13.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:12.819+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:13.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:12.819+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:13.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:12.819+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:13.654 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:23:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:23:13.363+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 30376 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:23:13.655 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:13.355+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:13.655 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:13.355+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:13.655 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:13.355+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:13.655 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:13.355+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:13.655 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:13.493+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:13.656 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:13.493+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:13.656 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:13.493+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:13.656 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:13.493+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:14.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:13.818+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:14.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:13.818+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:14.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:13.818+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:14.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:13.818+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:14.380+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:14.380+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:14.380+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:14.380+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:14.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:14.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:14.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:14.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:14.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:14.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:14.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:14.448+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:15.189 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:14.831+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:15.189 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:14.831+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:15.189 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:14.831+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:15.189 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:14.831+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:15.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:15.363+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:15.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:15.363+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:15.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:15.363+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:15.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:15.363+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:15.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:15.426+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:15.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:15.426+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:15.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:15.426+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:15.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:15.426+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:16.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:15.858+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:16.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:15.858+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:16.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:15.858+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:16.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:15.858+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:16.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:16.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:16.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:16.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:16.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:16.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:16.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:16.403+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:16.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:16.395+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:16.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:16.395+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:16.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:16.395+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:16.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:16.395+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:17.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:16.893+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:17.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:16.893+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:17.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:16.893+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:17.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:16.893+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:17.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:17.384+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:17.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:17.384+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:17.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:17.384+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:17.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:17.384+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:17.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:17.434+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:17.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:17.434+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:17.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:17.434+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:17.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:17.434+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:18.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:17.873+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:18.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:17.873+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:18.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:17.873+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:18.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:17.874+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:18.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:23:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:23:18.367+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 30459 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:23:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:18.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:18.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:18.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:18.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:18.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:18.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:18.463+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:18.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:18.394+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:18.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:18.394+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:18.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:18.394+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:18.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:18.394+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:19.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:18.879+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:19.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:18.879+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:19.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:18.879+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:19.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:18.879+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:19.392+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:19.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:19.392+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:19.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:19.392+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:19.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:19.392+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:19.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:19.509+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:19.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:19.509+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:19.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:19.509+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:19.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:19.509+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:20.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:19.917+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:20.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:19.917+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:20.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:19.917+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:20.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:19.917+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:20.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:20.497+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:20.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:20.497+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:20.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:20.497+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:20.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:20.497+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:20.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:20.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:20.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:20.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:20.757 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:20.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:20.757 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:20.399+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:20.954+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:20.954+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:20.954+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:20.954+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:21.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:21.455+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:21.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:21.455+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:21.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:21.455+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:21.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:21.455+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:21.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:21.377+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:21.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:21.377+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:21.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:21.377+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:21.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:21.377+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:22.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:21.990+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:22.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:21.990+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:22.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:21.990+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:22.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:21.990+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:22.414+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:22.414+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:22.414+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:22.414+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:22.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:22.343+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:22.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:22.343+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:22.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:22.343+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:22.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:22.343+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:23.339 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:23.013+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:23.339 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:23.013+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:23.339 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:23.013+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:23.339 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:23.014+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:23.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:23:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:23:23.372+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 30545 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:23:23.656 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:23.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:23.656 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:23.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:23.656 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:23.413+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:23.656 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:23.414+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:23.657 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:23.337+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:23.657 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:23.337+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:23.657 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:23.337+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:23.657 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:23.337+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:24.372 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:24.036+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:24.372 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:24.036+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:24.372 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:24.036+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:24.372 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:24.036+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:24.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:24.450+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:24.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:24.450+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:24.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:24.450+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:24.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:24.451+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:24.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:24.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:24.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:24.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:24.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:24.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:24.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:24.370+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:25.408 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:25.003+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:25.408 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:25.003+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:25.408 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:25.003+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:25.408 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:25.003+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:25.482+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:25.482+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:25.482+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:25.482+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:25.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:25.406+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:25.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:25.406+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:25.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:25.406+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:25.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:25.406+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:26.449 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:26.038+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:26.450 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:26.038+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:26.450 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:26.038+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:26.450 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:26.038+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:26.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:26.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:26.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:26.448+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:26.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:26.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:26.463+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:26.463+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:27.442 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:27.077+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:27.443 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:27.077+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:27.443 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:27.077+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:27.443 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:27.077+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:27.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:27.456+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:27.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:27.456+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:27.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:27.456+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:27.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:27.457+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:27.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:27.441+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:27.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:27.441+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:27.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:27.441+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:27.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:27.441+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:28.380 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:28.064+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:28.380 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:28.064+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:28.380 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:28.064+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:28.381 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:28.065+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:28.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:23:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:23:28.377+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 30628 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:23:28.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:28.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:28.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:28.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:28.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:28.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:28.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:28.473+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:28.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:28.428+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:28.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:28.428+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:28.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:28.428+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:28.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:28.429+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:29.408 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:29.065+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:29.408 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:29.065+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:29.409 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:29.065+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:29.409 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:29.066+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:29.475+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:29.475+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:29.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:29.475+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:29.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:29.476+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:29.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:29.406+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:29.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:29.406+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:29.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:29.406+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:29.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:29.406+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:30.383 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:30.116+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:30.384 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:30.116+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:30.384 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:30.116+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:30.384 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:30.116+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:30.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:30.381+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:30.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:30.381+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:30.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:30.381+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:30.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:30.381+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:30.472+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:30.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:30.472+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:30.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:30.472+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:30.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:30.472+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:31.422 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:31.110+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:31.422 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:31.110+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:31.422 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:31.110+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:31.422 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:31.110+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:31.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:31.420+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:31.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:31.420+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:31.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:31.420+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:31.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:31.420+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:31.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:31.426+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:31.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:31.426+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:31.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:31.426+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:31.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:31.426+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:32.373 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:32.063+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:32.374 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:32.063+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:32.374 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:32.063+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:32.374 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:32.063+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:32.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:32.372+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:32.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:32.372+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:32.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:32.372+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:32.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:32.372+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:32.469+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:32.469+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:32.469+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:32.469+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:33.353 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:33.053+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:33.353 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:33.053+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:33.354 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:33.053+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:33.354 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:33.053+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:33.621 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:23:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:23:33.383+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 30714 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:23:33.621 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:33.351+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:33.621 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:33.351+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:33.622 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:33.351+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:33.622 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:33.351+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:33.622 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:33.472+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:33.622 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:33.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:33.622 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:33.473+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:33.622 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:33.473+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:34.318 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:34.008+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:34.318 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:34.008+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:34.318 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:34.008+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:34.319 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:34.009+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:34.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:34.316+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:34.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:34.316+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:34.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:34.316+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:34.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:34.317+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:34.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:34.461+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:34.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:34.461+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:34.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:34.461+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:34.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:34.462+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:35.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:34.984+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:35.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:34.984+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:35.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:34.984+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:35.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:34.984+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:35.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:35.341+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:35.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:35.341+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:35.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:35.341+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:35.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:35.341+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:35.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:35.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:35.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:35.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:35.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:35.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:35.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:35.453+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:36.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:35.934+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:36.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:35.934+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:36.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:35.934+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:36.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:35.935+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:36.693 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:36.487+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:36.693 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:36.487+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:36.693 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:36.487+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:36.694 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:36.487+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:36.694 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:36.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:36.694 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:36.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:36.694 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:36.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:36.694 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:36.390+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:37.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:36.899+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:37.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:36.899+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:37.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:36.899+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:37.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:36.900+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:37.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:37.481+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:37.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:37.481+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:37.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:37.481+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:37.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:37.481+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:37.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:37.343+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:37.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:37.343+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:37.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:37.343+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:37.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:37.344+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:38.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:37.893+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:38.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:37.893+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:38.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:37.893+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:38.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:37.894+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:38.514 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:23:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:23:38.388+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 30797 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:23:38.514 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:38.489+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:38.514 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:38.489+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:38.514 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:38.489+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:38.514 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:38.489+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:38.515 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:38.384+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:38.515 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:38.384+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:38.515 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:38.384+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:38.515 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:38.384+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:39.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:38.938+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:39.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:38.938+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:39.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:38.938+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:39.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:38.938+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:39.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:39.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:39.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:39.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:39.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:39.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:39.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:39.464+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:39.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:39.362+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:39.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:39.362+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:39.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:39.362+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:39.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:39.362+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:40.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:39.974+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:40.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:39.974+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:40.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:39.974+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:40.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:39.974+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:40.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:40.401+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:40.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:40.401+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:40.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:40.401+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:40.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:40.401+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:40.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:40.469+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:40.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:40.469+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:40.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:40.469+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:40.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:40.469+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:41.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:40.991+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:41.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:40.991+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:41.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:40.991+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:41.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:40.992+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:41.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:41.467+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:41.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:41.467+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:41.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:41.467+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:41.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:41.468+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:41.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:41.411+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:41.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:41.411+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:41.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:41.411+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:41.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:41.411+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:42.409 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:42.033+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:42.410 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:42.033+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:42.410 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:42.033+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:42.410 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:42.034+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:42.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:42.509+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:42.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:42.509+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:42.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:42.509+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:42.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:42.509+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:42.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:42.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:42.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:42.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:42.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:42.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:42.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:42.408+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:43.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:43.049+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:43.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:43.049+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:43.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:43.049+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:43.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:43.050+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:43.657 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:23:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:23:43.394+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 30883 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:23:43.657 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:43.493+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:43.657 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:43.493+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:43.658 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:43.493+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:43.658 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:43.493+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:43.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:43.395+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:43.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:43.395+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:43.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:43.395+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:43.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:43.395+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:44.398 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:44.007+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:44.398 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:44.007+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:44.398 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:44.007+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:44.398 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:44.008+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:44.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:44.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:44.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:44.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:44.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:44.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:44.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:44.463+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:44.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:44.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:44.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:44.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:44.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:44.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:44.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:44.396+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:45.381 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:45.036+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:45.381 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:45.036+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:45.382 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:45.036+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:45.382 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:45.036+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:45.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:45.380+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:45.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:45.380+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:45.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:45.380+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:45.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:45.380+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:45.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:45.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:45.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:45.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:45.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:45.425+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:45.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:45.426+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:46.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:45.987+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:46.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:45.987+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:46.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:45.987+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:46.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:45.987+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:46.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:46.386+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:46.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:46.386+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:46.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:46.386+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:46.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:46.386+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:46.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:46.344+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:46.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:46.344+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:46.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:46.344+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:46.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:46.344+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:47.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:46.965+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:47.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:46.965+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:47.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:46.965+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:47.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:46.966+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:47.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:47.400+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:47.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:47.400+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:47.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:47.400+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:47.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:47.400+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:47.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:47.334+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:47.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:47.334+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:47.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:47.334+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:47.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:47.334+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:48.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:47.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:48.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:47.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:48.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:47.988+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:48.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:47.989+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:48.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:23:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:23:48.399+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 30966 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:23:48.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:48.378+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:48.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:48.378+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:48.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:48.378+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:48.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:48.378+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:48.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:48.437+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:48.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:48.437+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:48.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:48.437+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:48.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:48.438+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:49.340 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:49.032+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:49.341 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:49.032+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:49.341 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:49.032+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:49.341 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:49.032+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:49.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:49.338+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:49.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:49.338+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:49.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:49.338+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:49.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:49.339+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:49.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:49.447+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:49.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:49.447+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:49.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:49.447+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:49.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:49.448+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:50.323 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:50.035+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:50.323 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:50.035+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:50.323 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:50.035+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:50.324 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:50.035+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:50.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:50.321+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:50.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:50.321+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:50.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:50.321+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:50.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:50.322+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:50.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:50.426+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:50.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:50.426+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:50.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:50.426+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:50.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:50.427+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:51.339 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:51.059+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:51.339 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:51.059+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:51.339 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:51.059+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:51.339 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:51.059+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:51.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:51.337+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:51.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:51.337+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:51.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:51.337+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:51.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:51.338+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:51.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:51.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:51.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:51.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:51.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:51.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:51.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:51.429+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:52.347 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:52.019+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:52.347 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:52.019+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:52.348 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:52.019+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:52.348 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:52.019+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:52.427+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:52.427+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:52.427+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:52.427+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:52.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:52.346+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:52.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:52.346+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:52.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:52.346+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:52.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:52.346+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:53.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:52.973+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:53.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:52.973+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:53.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:52.973+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:53.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:52.973+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:53.658 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:23:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:23:53.404+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 31052 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:23:53.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:53.384+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:53.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:53.384+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:53.659 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:53.384+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:53.659 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:53.384+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:53.659 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:53.380+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:53.659 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:53.380+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:53.659 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:53.380+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:53.659 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:53.381+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:54.368 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:54.022+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:54.368 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:54.022+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:54.368 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:54.022+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:54.368 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:54.022+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:54.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:54.366+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:54.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:54.366+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:54.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:54.366+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:54.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:54.367+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:54.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:54.414+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:54.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:54.414+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:54.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:54.414+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:54.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:54.414+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:55.325 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:55.015+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:55.325 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:55.015+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:55.326 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:55.015+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:55.326 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:55.015+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:55.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:55.323+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:55.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:55.323+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:55.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:55.323+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:55.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:55.324+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:55.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:55.423+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:55.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:55.423+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:55.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:55.423+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:55.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:55.423+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:55.968+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:55.968+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:55.968+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:55.968+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:56.325+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:56.325+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:56.325+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:56.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:56.325+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:56.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:56.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:56.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:56.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:56.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:56.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:56.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:56.408+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:57.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:56.961+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:57.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:56.961+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:57.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:56.961+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:57.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:56.961+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:57.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:57.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:57.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:57.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:57.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:57.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:57.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:57.374+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:57.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:57.447+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:57.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:57.447+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:57.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:57.447+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:57.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:57.447+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:58.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:57.971+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:58.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:57.971+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:58.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:57.971+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:58.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:57.972+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:58.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:23:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:23:58.409+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 31135 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:23:58.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:58.461+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:58.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:58.461+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:58.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:58.461+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:58.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:58.461+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:58.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:58.330+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:58.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:58.330+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:58.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:58.330+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:58.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:58.330+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:58.968+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:58.968+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:58.968+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:23:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:58.968+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:59.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:59.294+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:59.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:59.294+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:59.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:59.294+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:23:59.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:23:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:23:59.294+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:23:59.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:59.500+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:59.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:59.500+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:59.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:59.500+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:23:59.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:23:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:23:59.500+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:00.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:59.946+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:00.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:59.946+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:00.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:59.946+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:00.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:23:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:23:59.946+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:00.537 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:00.270+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:00.537 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:00.270+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:00.538 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:00.270+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:00.538 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:00.271+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:00.920 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:00.536+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:00.920 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:00.536+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:00.920 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:00.536+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:00.921 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:00.536+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:01.233 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:00.918+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:01.233 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:00.918+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:01.234 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:00.918+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:01.234 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:00.919+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:01.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:01.231+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:01.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:01.231+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:01.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:01.231+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:01.502 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:01.232+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:01.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:01.500+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:01.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:01.500+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:01.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:01.500+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:01.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:01.500+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:02.204 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:01.963+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:02.205 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:01.963+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:02.205 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:01.963+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:02.205 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:01.964+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:02.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:02.203+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:02.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:02.203+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:02.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:02.203+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:02.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:02.203+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:02.953 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:02.509+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:02.953 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:02.509+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:02.954 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:02.509+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:02.954 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:02.509+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:03.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:02.952+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:03.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:02.952+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:03.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:02.952+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:03.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:02.952+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:03.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:03.156+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:03.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:03.156+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:03.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:03.156+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:03.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:03.156+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:03.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:24:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:24:03.414+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 31221 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:24:03.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:03.534+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:03.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:03.534+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:03.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:03.534+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:03.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:03.534+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:04.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:03.993+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:04.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:03.994+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:04.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:03.994+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:04.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:03.994+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:04.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:04.127+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:04.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:04.127+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:04.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:04.127+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:04.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:04.127+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:04.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:04.548+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:04.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:04.548+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:04.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:04.548+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:04.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:04.549+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:05.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:04.954+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:05.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:04.954+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:05.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:04.954+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:05.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:04.955+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:05.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:05.129+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:05.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:05.129+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:05.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:05.129+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:05.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:05.129+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:05.975 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:05.535+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:05.975 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:05.535+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:05.975 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:05.535+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:05.975 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:05.536+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:06.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:05.973+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:06.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:05.973+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:06.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:05.973+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:06.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:05.973+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:06.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:06.101+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:06.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:06.101+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:06.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:06.101+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:06.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:06.102+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:06.973 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:06.580+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:06.973 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:06.580+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:06.973 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:06.580+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:06.974 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:06.580+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:06.971+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:06.971+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:06.971+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:06.972+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:07.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:07.088+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:07.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:07.088+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:07.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:07.088+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:07.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:07.088+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:07.963 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:07.559+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:07.963 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:07.559+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:07.963 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:07.559+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:07.963 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:07.559+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:08.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:07.961+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:08.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:07.961+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:08.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:07.961+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:08.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:07.962+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:08.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:08.130+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:08.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:08.130+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:08.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:08.130+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:08.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:08.130+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:08.529 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:24:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:24:08.420+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 31305 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:24:08.951 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:08.528+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:08.951 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:08.528+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:08.951 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:08.528+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:08.951 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:08.528+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:08.949+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:08.949+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:08.949+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:09.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:08.950+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:09.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:09.154+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:09.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:09.154+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:09.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:09.154+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:09.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:09.154+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:09.906 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:09.512+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:09.907 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:09.512+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:09.907 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:09.512+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:09.907 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:09.512+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:10.182 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:09.905+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:10.183 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:09.905+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:10.183 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:09.905+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:10.183 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:09.905+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:10.468 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:10.181+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:10.468 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:10.181+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:10.468 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:10.181+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:10.468 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:10.181+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:10.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:10.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:10.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:10.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:10.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:10.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:10.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:10.466+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:11.200 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:10.935+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:11.200 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:10.935+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:11.201 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:10.935+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:11.201 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:10.935+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:11.498 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:11.199+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:11.498 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:11.199+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:11.498 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:11.199+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:11.498 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:11.199+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:11.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:11.496+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:11.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:11.496+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:11.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:11.496+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:11.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:11.496+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:12.228 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:11.976+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:12.228 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:11.976+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:12.228 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:11.976+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:12.229 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:11.977+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:12.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:12.226+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:12.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:12.226+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:12.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:12.226+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:12.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:12.227+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:12.951 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:12.539+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:12.951 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:12.539+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:12.951 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:12.539+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:12.952 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:12.540+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:13.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:12.950+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:13.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:12.950+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:13.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:12.950+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:13.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:12.950+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:13.530 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:24:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:24:13.425+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 31390 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:24:13.531 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:13.252+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:13.531 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:13.252+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:13.531 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:13.252+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:13.531 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:13.253+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:13.921 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:13.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:13.921 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:13.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:13.921 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:13.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:13.922 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:13.529+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:14.215 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:13.920+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:14.215 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:13.920+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:14.215 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:13.920+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:14.216 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:13.920+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:14.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:14.213+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:14.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:14.213+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:14.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:14.213+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:14.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:14.214+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:14.879 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:14.509+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:14.879 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:14.509+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:14.879 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:14.509+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:14.880 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:14.509+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:15.249 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:14.878+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:15.249 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:14.878+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:15.250 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:14.878+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:15.250 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:14.878+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:15.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:15.248+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:15.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:15.248+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:15.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:15.248+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:15.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:15.248+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:15.879 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:15.531+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:15.879 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:15.531+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:15.879 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:15.531+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:15.879 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:15.531+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:16.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:15.878+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:16.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:15.878+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:16.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:15.878+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:16.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:15.878+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:16.553 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:16.262+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:16.553 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:16.262+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:16.553 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:16.262+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:16.553 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:16.262+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:16.904 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:16.551+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:16.904 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:16.551+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:16.904 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:16.551+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:16.904 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:16.551+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:17.225 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:16.902+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:17.226 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:16.902+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:17.226 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:16.902+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:17.226 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:16.902+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:17.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:17.224+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:17.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:17.224+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:17.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:17.224+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:17.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:17.224+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:17.915 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:17.513+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:17.915 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:17.513+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:17.915 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:17.513+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:17.916 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:17.513+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:18.212 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:17.913+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:18.212 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:17.913+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:18.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:17.913+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:18.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:17.913+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:18.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:24:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:24:18.430+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 31474 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:24:18.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:18.211+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:18.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:18.211+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:18.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:18.211+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:18.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:18.211+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:18.892 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:18.525+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:18.892 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:18.525+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:18.892 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:18.525+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:18.892 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:18.525+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:19.250 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:18.890+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:19.250 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:18.890+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:19.250 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:18.890+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:19.250 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:18.891+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:19.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:19.248+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:19.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:19.248+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:19.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:19.248+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:19.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:19.248+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:19.913 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:19.509+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:19.913 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:19.509+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:19.914 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:19.509+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:19.914 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:19.509+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:20.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:19.912+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:20.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:19.912+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:20.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:19.912+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:20.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:19.912+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:20.551 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:20.274+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:20.552 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:20.274+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:20.552 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:20.274+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:20.552 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:20.274+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:20.871 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:20.550+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:20.871 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:20.550+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:20.871 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:20.550+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:20.871 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:20.550+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:20.869+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:20.869+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:20.869+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:20.869+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:21.583 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:21.277+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:21.583 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:21.277+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:21.583 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:21.277+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:21.583 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:21.277+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:21.851 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:21.582+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:21.852 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:21.582+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:21.852 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:21.582+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:21.852 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:21.582+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:22.231 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:21.850+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:22.231 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:21.850+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:22.231 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:21.850+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:22.231 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:21.850+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:22.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:22.228+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:22.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:22.228+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:22.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:22.228+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:22.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:22.229+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:22.810 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:22.547+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:22.811 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:22.547+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:22.811 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:22.547+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:22.811 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:22.547+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:23.206 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:22.809+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:23.206 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:22.809+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:23.207 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:22.809+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:23.207 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:22.809+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:23.461 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:24:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:24:23.436+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 31558 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:24:23.461 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:23.205+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:23.461 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:23.205+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:23.462 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:23.205+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:23.462 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:23.205+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:23.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:23.588+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:23.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:23.588+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:23.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:23.588+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:23.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:23.588+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:24.157 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:23.764+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:24.157 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:23.764+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:24.158 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:23.764+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:24.158 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:23.764+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:24.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:24.156+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:24.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:24.156+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:24.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:24.156+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:24.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:24.156+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:25.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:24.791+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:25.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:24.791+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:25.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:24.791+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:25.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:24.791+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:25.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:24.637+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:25.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:24.637+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:25.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:24.637+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:25.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:24.637+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:25.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:25.143+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:25.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:25.143+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:25.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:25.143+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:25.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:25.143+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:26.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:25.762+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:26.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:25.762+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:26.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:25.762+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:26.006 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:25.762+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:26.006 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:25.612+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:26.007 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:25.612+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:26.007 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:25.612+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:26.007 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:25.612+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:26.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:26.146+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:26.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:26.146+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:26.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:26.146+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:26.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:26.147+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:27.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:26.774+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:27.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:26.774+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:27.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:26.774+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:27.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:26.774+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:27.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:26.602+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:27.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:26.602+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:27.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:26.602+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:27.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:26.602+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:27.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:27.160+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:27.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:27.160+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:27.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:27.160+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:27.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:27.160+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:28.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:27.754+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:28.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:27.754+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:28.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:27.754+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:28.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:27.755+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:28.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:27.618+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:28.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:27.618+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:28.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:27.618+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:28.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:27.618+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:28.443 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:28.145+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:28.444 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:28.145+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:28.444 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:28.145+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:28.444 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:28.145+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:28.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:24:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:24:28.441+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 31643 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:24:28.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:28.629+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:28.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:28.629+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:28.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:28.629+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:28.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:28.629+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:29.128 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:28.761+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:29.128 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:28.761+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:29.129 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:28.761+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:29.129 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:28.762+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:29.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:29.126+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:29.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:29.126+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:29.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:29.126+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:29.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:29.127+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:30.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:29.779+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:30.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:29.779+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:30.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:29.779+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:30.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:29.779+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:30.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:29.614+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:30.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:29.614+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:30.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:29.614+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:30.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:29.614+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:30.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:30.147+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:30.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:30.147+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:30.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:30.147+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:30.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:30.147+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:31.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:30.828+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:31.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:30.828+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:31.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:30.828+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:31.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:30.829+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:31.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:30.621+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:31.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:30.621+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:31.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:30.621+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:31.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:30.621+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:31.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:31.147+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:31.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:31.147+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:31.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:31.147+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:31.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:31.147+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:31.988 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:31.810+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:31.989 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:31.810+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:31.989 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:31.810+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:31.989 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:31.810+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:31.989 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:31.621+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:31.989 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:31.621+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:31.990 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:31.621+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:31.990 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:31.621+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:32.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:32.181+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:32.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:32.181+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:32.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:32.181+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:32.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:32.181+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:33.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:32.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:33.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:32.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:33.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:32.802+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:33.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:32.802+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:33.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:32.588+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:33.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:32.588+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:33.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:32.588+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:33.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:32.588+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:33.448 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:33.176+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:33.449 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:33.176+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:33.449 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:33.176+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:33.449 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:33.176+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:33.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:24:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:24:33.447+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 31727 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:24:33.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:33.590+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:33.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:33.590+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:33.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:33.590+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:33.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:33.590+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:34.219 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:33.834+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:34.219 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:33.834+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:34.219 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:33.834+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:34.219 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:33.834+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:34.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:34.218+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:34.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:34.218+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:34.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:34.218+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:34.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:34.218+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:35.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:34.588+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:35.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:34.588+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:35.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:34.588+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:35.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:34.588+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:35.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:34.831+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:35.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:34.831+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:35.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:34.831+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:35.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:34.832+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:35.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:35.205+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:35.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:35.205+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:35.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:35.205+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:35.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:35.205+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:36.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:35.856+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:36.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:35.856+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:36.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:35.856+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:36.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:35.856+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:36.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:35.621+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:36.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:35.621+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:36.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:35.621+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:36.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:35.621+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:36.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:36.186+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:36.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:36.186+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:36.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:36.186+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:36.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:36.186+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:37.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:36.851+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:37.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:36.851+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:37.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:36.851+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:37.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:36.851+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:37.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:36.660+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:37.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:36.660+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:37.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:36.660+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:37.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:36.660+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:37.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:37.234+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:37.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:37.234+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:37.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:37.234+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:37.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:37.234+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:38.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:37.708+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:38.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:37.708+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:38.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:37.708+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:38.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:37.708+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:38.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:37.894+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:38.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:37.894+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:38.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:37.894+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:38.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:37.894+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:38.454 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:38.244+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:38.454 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:38.244+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:38.454 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:38.244+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:38.454 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:38.245+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:38.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:24:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:24:38.452+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 31806 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:24:38.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:38.692+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:38.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:38.692+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:38.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:38.692+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:38.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:38.692+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:39.240 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:38.864+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:39.241 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:38.864+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:39.241 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:38.864+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:39.241 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:38.864+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:39.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:39.238+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:39.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:39.238+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:39.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:39.238+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:39.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:39.238+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:40.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:39.828+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:40.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:39.828+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:40.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:39.828+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:40.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:39.828+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:40.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:39.700+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:40.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:39.700+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:40.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:39.700+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:40.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:39.700+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:40.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:40.224+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:40.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:40.224+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:40.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:40.224+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:40.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:40.224+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:41.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:40.799+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:41.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:40.799+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:41.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:40.799+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:41.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:40.800+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:41.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:40.674+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:41.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:40.674+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:41.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:40.674+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:41.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:40.675+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:41.651 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:41.258+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:41.651 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:41.258+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:41.651 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:41.258+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:41.651 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:41.258+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:42.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:41.800+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:42.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:41.800+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:42.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:41.800+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:42.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:41.800+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:42.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:41.648+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:42.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:41.648+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:42.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:41.648+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:42.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:41.648+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:42.675 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:42.281+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:42.675 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:42.281+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:42.676 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:42.281+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:42.676 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:42.281+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:43.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:42.818+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:43.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:42.818+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:43.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:42.818+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:43.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:42.819+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:43.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:42.673+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:43.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:42.673+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:43.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:42.673+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:43.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:42.673+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:43.584 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:24:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:24:43.456+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 31884 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:24:43.585 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:43.251+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:43.585 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:43.251+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:43.585 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:43.251+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:43.585 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:43.252+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:44.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:43.806+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:44.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:43.806+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:44.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:43.806+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:44.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:43.806+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:44.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:43.667+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:44.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:43.667+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:44.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:43.667+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:44.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:43.667+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:44.650 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:44.284+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:44.650 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:44.284+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:44.651 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:44.284+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:44.651 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:44.285+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:44.781+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:44.781+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:44.781+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:45.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:44.781+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:45.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:44.648+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:45.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:44.648+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:45.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:44.648+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:45.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:44.648+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:45.242+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:45.242+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:45.242+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:45.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:45.243+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:46.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:45.770+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:46.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:45.770+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:46.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:45.770+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:46.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:45.770+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:46.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:45.610+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:46.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:45.610+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:46.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:45.610+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:46.005 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:45.610+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:46.621 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:46.270+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:46.621 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:46.270+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:46.621 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:46.270+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:46.622 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:46.270+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:47.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:46.769+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:47.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:46.769+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:47.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:46.769+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:47.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:46.769+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:47.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:46.619+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:47.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:46.619+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:47.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:46.619+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:47.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:46.619+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:47.604 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:47.272+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:47.605 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:47.272+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:47.605 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:47.272+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:47.605 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:47.272+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:47.759+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:47.759+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:47.759+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:48.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:47.759+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:48.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:47.602+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:48.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:47.602+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:48.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:47.602+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:48.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:47.602+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:48.588 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:24:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:24:48.462+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 31960 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:24:48.589 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:48.298+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:48.589 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:48.298+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:48.589 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:48.298+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:48.589 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:48.299+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:49.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:48.600+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:49.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:48.600+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:49.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:48.600+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:49.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:48.600+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:49.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:48.789+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:49.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:48.789+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:49.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:48.789+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:49.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:48.790+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:49.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:49.313+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:49.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:49.313+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:49.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:49.313+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:49.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:49.313+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:49.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:49.552+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:49.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:49.552+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:49.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:49.552+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:49.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:49.553+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:50.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:49.805+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:50.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:49.805+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:50.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:49.805+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:50.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:49.806+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:50.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:50.510+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:50.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:50.510+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:50.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:50.510+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:50.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:50.510+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:50.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:50.283+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:50.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:50.283+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:50.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:50.283+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:50.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:50.283+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:51.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:50.789+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:51.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:50.789+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:51.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:50.789+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:51.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:50.789+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:51.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:51.327+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:51.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:51.327+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:51.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:51.327+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:51.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:51.327+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:51.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:51.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:51.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:51.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:51.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:51.529+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:51.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:51.529+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:52.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:51.787+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:52.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:51.787+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:52.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:51.787+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:52.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:51.787+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:52.567 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:52.287+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:52.567 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:52.287+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:52.567 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:52.287+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:52.567 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:52.287+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:53.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:52.811+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:53.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:52.811+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:53.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:52.811+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:53.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:52.811+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:53.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:52.565+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:53.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:52.565+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:53.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:52.565+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:53.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:52.565+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:53.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:24:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:24:53.467+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 32035 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:24:53.596 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:53.276+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:53.597 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:53.276+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:53.597 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:53.276+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:53.597 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:53.276+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:53.597 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:53.516+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:53.598 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:53.516+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:53.598 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:53.516+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:53.598 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:53.516+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:54.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:53.813+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:54.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:53.813+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:54.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:53.813+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:54.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:53.813+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:54.250+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:54.250+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:54.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:54.250+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:54.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:54.250+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:54.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:54.494+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:54.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:54.494+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:54.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:54.494+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:54.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:54.494+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:55.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:54.863+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:55.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:54.863+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:55.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:54.863+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:55.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:54.863+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:55.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:55.287+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:55.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:55.287+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:55.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:55.287+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:55.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:55.287+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:55.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:55.446+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:55.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:55.446+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:55.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:55.446+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:55.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:55.446+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:55.900+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:55.900+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:55.900+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:56.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:55.900+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:56.329+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:56.329+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:56.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:56.329+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:56.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:56.330+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:56.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:56.463+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:56.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:56.463+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:56.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:56.463+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:56.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:56.464+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:57.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:56.865+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:57.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:56.865+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:57.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:56.865+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:57.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:56.865+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:57.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:57.313+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:57.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:57.313+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:57.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:57.313+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:57.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:57.313+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:57.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:57.484+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:57.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:57.484+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:57.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:57.484+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:57.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:57.484+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:58.256 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:57.878+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:58.257 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:57.878+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:58.257 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:57.878+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:58.257 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:57.879+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:58.599 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:24:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:24:58.473+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 32114 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:24:58.599 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:58.325+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:58.599 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:58.325+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:58.599 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:58.325+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:58.599 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:58.326+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:58.600 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:58.517+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:58.600 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:58.517+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:58.600 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:58.517+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:58.600 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:58.517+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:59.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:58.847+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:59.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:58.847+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:59.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:58.847+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:24:59.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:58.848+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:59.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:59.497+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:59.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:59.497+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:59.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:59.497+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:24:59.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:24:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:24:59.497+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:24:59.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:59.340+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:59.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:59.340+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:59.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:59.340+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:24:59.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:24:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:24:59.340+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:00.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:59.862+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:00.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:59.862+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:00.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:59.862+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:00.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:24:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:24:59.863+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:00.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:00.537+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:00.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:00.537+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:00.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:00.537+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:00.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:00.538+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:00.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:00.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:00.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:00.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:00.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:00.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:00.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:00.390+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:01.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:00.875+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:01.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:00.875+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:01.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:00.875+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:01.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:00.876+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:01.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:01.560+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:01.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:01.560+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:01.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:01.560+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:01.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:01.561+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:01.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:01.347+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:01.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:01.347+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:01.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:01.347+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:01.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:01.347+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:02.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:01.859+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:02.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:01.859+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:02.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:01.859+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:02.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:01.860+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:02.609 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:02.308+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:02.609 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:02.308+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:02.610 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:02.308+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:02.610 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:02.308+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:03.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:02.810+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:03.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:02.810+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:03.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:02.810+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:03.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:02.811+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:03.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:02.607+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:03.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:02.607+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:03.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:02.607+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:03.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:02.608+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:03.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:25:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:25:03.478+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 32195 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:25:03.594 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:03.310+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:03.594 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:03.310+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:03.595 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:03.310+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:03.595 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:03.311+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:04.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:03.592+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:04.003 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:03.592+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:04.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:03.592+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:04.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:03.592+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:04.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:03.799+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:04.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:03.799+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:04.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:03.799+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:04.005 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:03.800+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:04.580 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:04.310+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:04.580 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:04.310+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:04.581 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:04.310+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:04.581 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:04.310+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:05.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:04.755+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:05.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:04.755+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:05.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:04.755+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:05.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:04.756+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:05.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:04.578+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:05.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:04.578+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:05.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:04.578+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:05.004 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:04.579+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:05.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:05.549+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:05.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:05.549+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:05.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:05.549+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:05.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:05.549+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:05.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:05.341+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:05.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:05.341+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:05.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:05.341+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:05.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:05.341+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:06.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:05.805+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:06.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:05.805+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:06.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:05.805+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:06.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:05.806+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:06.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:06.504+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:06.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:06.504+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:06.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:06.504+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:06.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:06.504+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:06.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:06.389+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:06.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:06.389+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:06.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:06.389+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:06.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:06.389+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:06.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:07.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:06.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:07.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:06.801+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:07.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:06.801+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:07.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:07.540+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:07.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:07.541+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:07.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:07.541+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:07.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:07.541+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:07.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:07.369+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:07.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:07.369+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:07.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:07.369+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:07.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:07.369+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:08.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:07.832+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:08.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:07.832+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:08.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:07.832+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:08.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:07.833+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:08.512 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:25:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:25:08.484+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 32280 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:25:08.512 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:08.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:08.512 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:08.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:08.512 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:08.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:08.513 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:08.375+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:08.843 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:08.558+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:08.843 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:08.558+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:08.843 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:08.558+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:08.843 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:08.558+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:08.841+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:08.841+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:08.841+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:09.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:08.841+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:09.346+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:09.346+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:09.346+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:09.346+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:09.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:09.518+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:09.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:09.518+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:09.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:09.518+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:09.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:09.518+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:10.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:09.855+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:10.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:09.855+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:10.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:09.855+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:10.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:09.856+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:10.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:10.395+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:10.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:10.395+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:10.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:10.395+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:10.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:10.396+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:10.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:10.495+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:10.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:10.495+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:10.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:10.495+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:10.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:10.495+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:11.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:10.842+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:11.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:10.842+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:11.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:10.842+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:11.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:10.842+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:11.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:11.372+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:11.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:11.372+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:11.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:11.372+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:11.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:11.372+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:11.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:11.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:11.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:11.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:11.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:11.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:11.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:11.474+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:12.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:11.849+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:12.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:11.849+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:12.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:11.849+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:12.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:11.849+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:12.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:12.395+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:12.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:12.395+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:12.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:12.395+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:12.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:12.395+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:12.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:12.504+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:12.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:12.504+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:12.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:12.504+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:12.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:12.505+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:13.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:12.853+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:13.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:12.853+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:13.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:12.853+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:13.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:12.854+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:13.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:25:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:25:13.489+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 32364 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:25:13.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:13.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:13.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:13.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:13.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:13.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:13.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:13.423+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:13.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:13.489+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:13.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:13.489+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:13.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:13.489+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:13.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:13.489+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:14.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:13.851+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:14.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:13.851+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:14.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:13.851+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:14.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:13.851+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:14.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:14.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:14.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:14.440+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:14.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:14.481+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:14.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:14.481+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:14.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:14.481+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:14.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:14.481+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:15.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:14.896+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:15.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:14.896+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:15.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:14.896+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:15.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:14.896+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:15.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:15.467+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:15.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:15.467+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:15.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:15.467+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:15.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:15.468+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:15.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:15.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:15.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:15.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:15.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:15.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:15.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:15.465+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:16.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:15.859+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:16.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:15.859+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:16.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:15.859+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:16.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:15.860+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:16.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:16.449+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:16.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:16.449+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:16.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:16.449+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:16.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:16.450+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:16.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:16.493+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:16.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:16.493+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:16.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:16.493+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:16.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:16.493+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:17.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:16.829+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:17.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:16.829+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:17.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:16.829+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:17.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:16.830+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:17.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:17.480+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:17.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:17.480+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:17.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:17.480+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:17.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:17.480+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:17.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:17.414+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:17.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:17.414+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:17.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:17.414+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:17.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:17.414+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:18.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:17.783+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:18.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:17.783+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:18.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:17.783+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:18.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:17.784+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:18.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:25:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:25:18.494+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 32447 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:25:18.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:18.506+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:18.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:18.506+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:18.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:18.506+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:18.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:18.507+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:18.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:18.439+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:18.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:18.439+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:18.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:18.439+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:18.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:18.440+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:19.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:18.770+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:19.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:18.770+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:19.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:18.770+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:19.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:18.771+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:19.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:19.520+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:19.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:19.520+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:19.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:19.520+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:19.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:19.520+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:19.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:19.441+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:19.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:19.441+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:19.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:19.441+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:19.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:19.441+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:20.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:19.786+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:20.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:19.786+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:20.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:19.786+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:20.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:19.786+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:20.509+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:20.509+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:20.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:20.509+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:20.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:20.510+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:20.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:20.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:20.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:20.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:20.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:20.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:20.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:20.432+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:20.797+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:20.797+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:21.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:20.797+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:21.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:20.797+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:21.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:21.490+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:21.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:21.490+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:21.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:21.490+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:21.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:21.490+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:21.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:21.433+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:21.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:21.433+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:21.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:21.433+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:21.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:21.433+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:22.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:21.819+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:22.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:21.819+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:22.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:21.819+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:22.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:21.820+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:22.459+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:22.459+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:22.459+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:22.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:22.459+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:22.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:22.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:22.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:22.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:22.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:22.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:22.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:22.480+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:23.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:22.820+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:23.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:22.820+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:23.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:22.820+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:23.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:22.820+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:23.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:25:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:25:23.500+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 32531 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:25:23.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:23.523+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:23.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:23.523+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:23.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:23.523+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:23.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:23.523+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:23.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:23.477+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:23.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:23.477+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:23.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:23.477+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:23.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:23.478+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:24.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:23.857+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:24.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:23.857+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:24.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:23.857+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:24.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:23.857+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:24.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:24.499+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:24.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:24.499+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:24.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:24.499+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:24.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:24.500+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:24.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:24.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:24.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:24.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:24.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:24.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:24.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:24.479+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:25.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:24.825+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:25.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:24.825+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:25.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:24.825+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:25.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:24.825+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:25.502+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:25.502+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:25.502+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:25.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:25.503+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:25.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:25.439+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:25.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:25.439+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:25.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:25.439+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:25.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:25.439+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:26.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:25.798+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:26.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:25.798+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:26.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:25.798+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:26.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:25.798+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:26.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:26.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:26.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:26.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:26.551+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:26.551+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:26.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:26.485+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:26.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:26.485+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:26.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:26.485+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:26.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:26.486+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:27.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:26.773+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:27.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:26.773+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:27.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:26.773+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:27.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:26.774+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:27.566+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:27.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:27.566+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:27.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:27.566+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:27.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:27.567+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:27.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:27.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:27.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:27.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:27.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:27.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:27.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:27.448+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:28.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:27.796+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:28.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:27.796+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:28.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:27.796+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:28.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:27.796+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:28.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:25:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:25:28.505+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 32615 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:25:28.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:28.603+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:28.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:28.603+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:28.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:28.603+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:28.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:28.603+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:28.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:28.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:28.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:28.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:28.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:28.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:28.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:28.474+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:29.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:28.804+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:29.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:28.804+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:29.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:28.804+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:29.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:28.805+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:29.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:29.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:29.584+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:29.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:29.585+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:29.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:29.500+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:29.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:29.500+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:29.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:29.500+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:29.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:29.500+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:30.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:29.826+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:30.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:29.826+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:30.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:29.826+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:30.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:29.826+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:30.626+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:30.626+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:30.626+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:30.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:30.626+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:30.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:30.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:30.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:30.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:30.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:30.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:30.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:30.466+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:31.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:30.792+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:31.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:30.792+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:31.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:30.792+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:31.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:30.793+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:31.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:31.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:31.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:31.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:31.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:31.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:31.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:31.662+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:31.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:31.427+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:31.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:31.427+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:31.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:31.427+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:31.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:31.427+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:32.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:31.800+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:32.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:31.800+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:32.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:31.800+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:32.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:31.800+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:32.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:32.477+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:32.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:32.477+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:32.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:32.477+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:32.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:32.477+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:32.678+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:32.678+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:32.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:32.678+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:32.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:32.678+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:33.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:32.768+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:33.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:32.768+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:33.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:32.768+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:33.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:32.768+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:33.700 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:25:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:25:33.511+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 32699 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:25:33.700 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:33.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:33.701 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:33.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:33.701 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:33.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:33.701 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:33.433+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:34.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:33.698+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:34.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:33.698+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:34.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:33.698+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:34.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:33.699+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:34.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:33.729+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:34.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:33.729+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:34.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:33.729+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:34.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:33.729+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:34.709 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:34.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:34.710 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:34.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:34.710 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:34.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:34.710 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:34.408+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:35.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:34.708+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:35.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:34.708+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:35.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:34.708+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:35.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:34.708+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:35.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:34.721+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:35.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:34.721+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:35.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:34.721+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:35.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:34.721+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:35.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:35.440+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:35.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:35.440+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:35.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:35.440+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:35.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:35.440+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:35.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:35.678+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:35.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:35.678+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:35.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:35.678+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:35.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:35.679+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:35.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:35.678+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:35.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:35.678+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:35.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:35.678+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:35.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:35.678+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:36.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:36.452+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:36.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:36.452+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:36.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:36.452+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:36.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:36.452+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:36.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:36.640+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:36.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:36.640+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:36.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:36.640+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:36.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:36.641+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:36.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:36.635+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:36.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:36.635+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:36.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:36.635+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:36.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:36.636+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:37.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:37.464+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:37.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:37.464+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:37.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:37.464+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:37.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:37.464+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:37.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:37.599+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:37.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:37.599+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:37.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:37.599+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:37.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:37.600+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:37.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:37.683+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:37.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:37.683+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:37.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:37.683+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:37.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:37.683+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:38.712 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:25:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:25:38.516+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 32784 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:25:38.712 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:38.620+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:38.713 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:38.620+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:38.713 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:38.620+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:38.713 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:38.620+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:38.713 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:38.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:38.713 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:38.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:38.713 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:38.448+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:38.714 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:38.448+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:39.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:38.711+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:39.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:38.711+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:39.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:38.711+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:39.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:38.711+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:39.730 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:39.598+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:39.731 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:39.598+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:39.731 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:39.598+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:39.731 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:39.598+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:39.731 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:39.426+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:39.731 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:39.426+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:39.731 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:39.426+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:39.732 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:39.426+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:39.728+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:39.728+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:40.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:39.728+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:40.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:39.729+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:40.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:40.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:40.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:40.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:40.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:40.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:40.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:40.465+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:40.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:40.638+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:40.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:40.638+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:40.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:40.638+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:40.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:40.638+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:40.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:40.697+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:40.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:40.697+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:40.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:40.697+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:40.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:40.698+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:41.690 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:41.604+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:41.690 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:41.604+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:41.690 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:41.604+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:41.691 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:41.605+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:41.691 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:41.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:41.691 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:41.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:41.691 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:41.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:41.691 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:41.416+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:42.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:41.688+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:42.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:41.688+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:42.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:41.688+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:42.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:41.689+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:42.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:42.442+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:42.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:42.442+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:42.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:42.442+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:42.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:42.442+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:42.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:42.565+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:42.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:42.565+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:42.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:42.565+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:42.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:42.565+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:42.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:42.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:42.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:42.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:42.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:42.662+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:42.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:42.663+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:43.668 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:25:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:25:43.522+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 32868 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:25:43.668 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:43.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:43.668 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:43.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:43.668 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:43.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:43.668 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:43.398+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:43.669 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:43.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:43.669 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:43.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:43.669 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:43.563+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:43.669 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:43.564+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:44.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:43.708+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:44.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:43.708+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:44.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:43.708+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:44.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:43.709+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:44.715 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:44.591+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:44.716 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:44.591+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:44.716 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:44.591+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:44.716 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:44.592+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:44.716 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:44.371+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:44.716 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:44.371+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:44.716 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:44.371+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:44.716 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:44.371+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:45.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:44.714+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:45.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:44.714+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:45.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:44.714+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:45.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:44.714+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:45.717 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:45.409+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:45.717 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:45.409+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:45.717 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:45.409+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:45.717 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:45.409+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:45.718 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:45.547+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:45.718 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:45.547+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:45.718 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:45.547+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:45.718 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:45.547+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:46.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:45.734+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:46.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:45.734+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:46.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:45.734+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:46.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:45.734+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:46.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:46.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:46.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:46.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:46.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:46.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:46.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:46.432+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:46.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:46.557+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:46.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:46.557+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:46.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:46.557+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:46.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:46.557+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:47.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:46.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:47.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:46.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:47.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:46.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:47.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:46.753+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:47.741 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:47.425+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:47.741 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:47.425+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:47.741 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:47.425+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:47.742 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:47.426+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:47.742 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:47.539+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:47.742 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:47.539+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:47.742 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:47.539+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:47.743 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:47.540+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:48.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:47.739+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:48.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:47.739+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:48.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:47.739+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:48.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:47.739+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:48.652 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:25:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:25:48.527+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 32954 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:25:48.653 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:48.542+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:48.653 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:48.542+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:48.653 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:48.542+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:48.653 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:48.543+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:48.654 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:48.378+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:48.654 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:48.378+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:48.654 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:48.378+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:48.654 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:48.378+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:49.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:48.715+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:49.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:48.715+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:49.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:48.715+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:49.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:48.715+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:49.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:49.530+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:49.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:49.530+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:49.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:49.530+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:49.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:49.530+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:49.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:49.341+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:49.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:49.341+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:49.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:49.341+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:49.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:49.341+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:50.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:49.765+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:50.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:49.765+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:50.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:49.765+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:50.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:49.765+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:50.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:50.502+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:50.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:50.502+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:50.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:50.502+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:50.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:50.502+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:50.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:50.370+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:50.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:50.370+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:50.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:50.370+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:50.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:50.370+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:51.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:50.782+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:51.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:50.782+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:51.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:50.782+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:51.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:50.782+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:51.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:51.456+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:51.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:51.456+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:51.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:51.456+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:51.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:51.456+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:51.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:51.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:51.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:51.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:51.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:51.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:51.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:51.399+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:52.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:51.802+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:52.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:51.802+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:52.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:51.802+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:52.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:51.802+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:52.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:52.471+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:52.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:52.471+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:52.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:52.471+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:52.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:52.471+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:52.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:52.439+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:52.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:52.439+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:52.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:52.439+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:52.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:52.440+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:53.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:52.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:53.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:52.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:53.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:52.796+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:53.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:52.796+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:53.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:25:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:25:53.533+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 33038 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:25:53.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:53.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:53.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:53.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:53.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:53.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:53.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:53.451+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:53.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:53.417+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:53.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:53.417+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:53.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:53.417+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:53.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:53.417+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:54.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:53.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:54.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:53.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:54.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:53.776+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:54.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:53.776+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:54.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:54.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:54.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:54.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:54.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:54.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:54.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:54.455+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:54.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:54.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:54.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:54.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:54.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:54.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:54.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:54.490+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:55.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:54.757+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:55.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:54.757+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:55.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:54.757+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:55.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:54.758+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:55.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:55.443+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:55.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:55.443+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:55.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:55.443+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:55.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:55.443+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:55.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:55.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:55.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:55.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:55.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:55.490+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:55.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:55.490+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:56.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:55.798+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:56.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:55.798+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:56.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:55.798+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:56.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:55.798+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:56.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:56.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:56.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:56.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:56.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:56.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:56.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:56.409+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:56.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:56.529+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:56.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:56.529+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:56.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:56.529+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:56.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:56.529+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:57.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:56.828+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:57.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:56.828+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:57.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:56.828+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:57.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:56.829+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:57.301 ERROR:paramiko.transport:Socket exception: No route to host (113) 2023-12-16T21:25:57.308 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T21:25:57.309 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2023-12-16T21:25:57.310 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2023-12-16T21:25:57.328 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-12-16T21:25:57.341 INFO:tasks.cephadm:Teardown begin 2023-12-16T21:25:57.341 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1794, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2023-12-16T21:25:57.346 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-12-16T21:25:57.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:25:57.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:25:57.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:57.388+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:57.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:57.388+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:57.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:57.388+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:57.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:57.388+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:57.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:57.495+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:57.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:57.495+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:57.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:57.495+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:57.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:57.495+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:58.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:57.819+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:58.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:57.819+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:58.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:57.819+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:58.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:57.819+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:58.669 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:25:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:25:58.538+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 33123 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:25:58.670 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:58.387+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:58.670 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:58.387+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:58.670 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:58.387+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:58.670 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:58.387+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:58.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:58.467+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:58.671 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:58.467+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:58.671 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:58.467+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:58.671 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:58.468+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:59.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:58.803+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:59.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:58.803+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:59.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:58.803+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:25:59.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:58.803+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:59.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:59.391+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:59.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:59.391+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:59.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:59.391+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:25:59.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:25:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:25:59.392+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:25:59.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:59.450+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:59.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:59.450+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:59.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:59.450+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:25:59.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:25:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:25:59.450+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:00.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:59.813+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:00.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:59.813+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:00.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:59.813+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:00.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:25:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:25:59.814+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:00.465 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:26:00.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:00.493+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:00.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:00.493+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:00.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:00.493+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:00.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:00.493+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:00.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:00.403+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:00.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:00.403+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:00.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:00.403+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:00.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:00.403+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:01.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:00.800+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:01.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:00.800+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:01.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:00.800+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:01.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:00.800+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:01.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:01.468+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:01.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:01.468+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:01.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:01.468+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:01.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:01.468+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:01.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:01.433+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:01.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:01.433+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:01.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:01.433+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:01.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:01.434+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:02.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:01.753+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:02.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:01.753+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:02.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:01.753+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:02.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:01.753+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:02.737 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:02.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:02.737 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:02.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:02.737 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:02.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:02.737 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:02.396+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:02.737 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:02.510+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:02.738 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:02.510+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:02.738 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:02.510+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:02.738 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:02.510+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:03.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:02.734+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:03.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:02.734+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:03.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:02.734+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:03.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:02.734+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:03.669 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:26:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:26:03.543+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 33207 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:26:03.670 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:03.365+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:03.670 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:03.365+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:03.670 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:03.365+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:03.670 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:03.365+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:03.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:03.506+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:03.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:03.506+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:03.671 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:03.506+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:03.671 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:03.506+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:04.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:03.715+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:04.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:03.715+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:04.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:03.715+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:04.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:03.716+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:04.359+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:04.359+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:04.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:04.359+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:04.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:04.359+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:05.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:04.535+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:05.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:04.535+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:05.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:04.535+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:05.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:04.535+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:05.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:04.756+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:05.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:04.756+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:05.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:04.756+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:05.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:04.756+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:05.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:05.529+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:05.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:05.529+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:05.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:05.529+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:05.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:05.529+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:05.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:05.372+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:05.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:05.372+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:05.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:05.372+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:05.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:05.372+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:06.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:05.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:06.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:05.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:06.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:05.752+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:06.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:05.753+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:06.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:06.526+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:06.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:06.526+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:06.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:06.526+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:06.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:06.526+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:06.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:06.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:06.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:06.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:06.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:06.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:06.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:06.369+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:07.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:06.762+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:07.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:06.762+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:07.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:06.762+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:07.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:06.762+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:07.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:07.511+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:07.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:07.511+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:07.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:07.511+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:07.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:07.511+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:07.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:07.387+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:07.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:07.387+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:07.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:07.387+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:07.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:07.387+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:08.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:07.793+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:08.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:07.793+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:08.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:07.793+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:08.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:07.794+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:08.552 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:08.400+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:08.552 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:08.400+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:08.552 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:08.400+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:08.552 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:08.400+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:08.553 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:08.467+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:08.553 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:08.467+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:08.553 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:08.467+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:08.553 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:08.468+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:09.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:26:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:26:08.549+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 33293 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:26:09.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:08.753+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:09.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:08.753+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:09.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:08.753+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:09.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:08.754+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:09.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:26:09.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:26:09.720 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:09.418+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:09.720 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:09.418+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:09.720 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:09.418+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:09.721 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:09.418+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:09.721 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:09.491+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:09.721 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:09.491+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:09.721 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:09.491+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:09.722 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:09.492+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:10.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:09.717+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:10.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:09.717+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:10.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:09.717+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:10.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:09.717+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:10.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:10.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:10.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:10.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:10.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:10.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:10.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:10.540+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:10.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:10.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:10.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:10.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:10.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:10.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:10.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:10.432+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:11.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:10.753+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:11.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:10.753+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:11.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:10.753+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:11.254 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:10.753+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:11.734 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:11.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:11.735 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:11.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:11.735 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:11.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:11.735 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:11.577+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:11.735 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:11.397+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:11.735 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:11.397+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:11.735 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:11.397+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:11.736 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:11.397+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:12.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:11.732+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:12.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:11.732+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:12.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:11.732+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:12.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:11.732+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:12.529 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:26:12.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:12.414+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:12.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:12.414+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:12.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:12.414+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:12.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:12.414+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:12.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:12.595+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:12.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:12.595+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:12.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:12.595+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:12.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:12.596+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:13.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:12.750+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:13.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:12.750+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:13.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:12.750+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:13.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:12.750+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:13.669 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:26:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:26:13.554+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 33376 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:26:13.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:13.611+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:13.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:13.611+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:13.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:13.611+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:13.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:13.611+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:13.671 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:13.375+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:13.671 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:13.375+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:13.671 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:13.375+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:13.671 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:13.375+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:14.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:13.715+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:14.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:13.715+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:14.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:13.715+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:14.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:13.715+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:14.661 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:14.334+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:14.662 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:14.334+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:14.662 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:14.334+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:14.662 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:14.334+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:15.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:14.694+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:15.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:14.694+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:15.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:14.694+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:15.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:14.694+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:15.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:14.659+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:15.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:14.659+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:15.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:14.659+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:15.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:14.659+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:15.676 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:15.328+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:15.676 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:15.328+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:15.676 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:15.328+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:15.676 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:15.328+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:15.681+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:15.681+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:16.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:15.681+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:16.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:15.682+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:16.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:15.673+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:16.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:15.673+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:16.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:15.673+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:16.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:15.673+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:16.638 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:16.375+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:16.639 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:16.375+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:16.639 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:16.375+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:16.639 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:16.375+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:17.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:16.649+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:17.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:16.649+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:17.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:16.649+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:17.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:16.650+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:17.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:16.637+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:17.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:16.637+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:17.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:16.637+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:17.005 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:16.637+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:17.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:17.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:17.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:17.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:17.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:17.633+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:17.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:17.633+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:17.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:17.645+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:17.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:17.645+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:17.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:17.645+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:17.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:17.646+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:17.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:17.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:17.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:17.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:17.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:17.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:17.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:17.399+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:18.680 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:26:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:26:18.561+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 33462 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:26:18.680 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:18.645+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:18.680 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:18.645+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:18.680 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:18.645+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:18.680 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:18.646+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:18.681 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:18.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:18.681 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:18.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:18.681 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:18.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:18.681 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:18.416+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:19.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:18.678+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:19.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:18.678+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:19.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:18.678+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:19.004 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:18.678+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:19.700 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:19.613+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:19.700 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:19.613+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:19.700 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:19.613+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:19.700 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:19.614+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:19.701 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:19.445+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:19.701 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:19.445+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:19.701 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:19.445+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:19.701 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:19.445+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:19.698+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:19.698+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:19.698+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:20.003 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:19.698+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:20.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:20.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:20.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:20.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:20.665+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:20.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:20.666+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:20.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:20.425+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:20.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:20.425+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:20.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:20.425+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:20.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:20.426+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:20.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:20.589+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:20.756 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:20.589+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:20.756 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:20.589+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:20.756 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:20.590+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:21.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:21.685+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:21.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:21.685+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:21.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:21.685+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:21.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:21.686+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:21.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:21.475+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:21.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:21.475+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:21.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:21.475+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:21.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:21.475+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:21.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:21.610+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:21.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:21.610+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:21.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:21.610+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:21.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:21.610+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:22.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:22.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:22.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:22.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:22.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:22.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:22.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:22.577+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:22.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:22.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:22.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:22.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:22.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:22.642+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:22.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:22.643+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:22.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:22.447+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:22.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:22.447+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:22.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:22.447+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:22.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:22.447+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:23.693 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:26:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:26:23.566+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 33546 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:26:23.694 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:23.593+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:23.694 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:23.593+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:23.694 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:23.593+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:23.694 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:23.593+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:23.694 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:23.668+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:23.695 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:23.668+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:23.695 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:23.668+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:23.695 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:23.668+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:23.695 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:23.427+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:23.695 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:23.427+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:23.695 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:23.427+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:23.695 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:23.428+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:24.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:26:24.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:26:24.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:24.549+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:24.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:24.549+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:24.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:24.549+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:24.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:24.550+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:24.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:24.622+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:24.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:24.622+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:24.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:24.622+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:24.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:24.622+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:24.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:24.428+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:24.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:24.428+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:24.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:24.428+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:24.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:24.429+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:25.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:25.542+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:25.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:25.542+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:25.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:25.542+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:25.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:25.543+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:25.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:25.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:25.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:25.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:25.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:25.614+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:25.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:25.614+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:25.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:25.402+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:25.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:25.403+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:25.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:25.403+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:25.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:25.403+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:26.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:26.546+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:26.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:26.546+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:26.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:26.546+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:26.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:26.547+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:26.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:26.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:26.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:26.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:26.610+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:26.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:26.611+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:26.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:26.376+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:26.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:26.376+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:26.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:26.376+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:26.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:26.376+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:27.601 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:26:27.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:27.543+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:27.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:27.543+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:27.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:27.543+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:27.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:27.543+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:27.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:27.583+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:27.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:27.583+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:27.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:27.583+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:27.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:27.584+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:27.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:27.371+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:27.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:27.371+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:27.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:27.371+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:27.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:27.372+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:28.698 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:26:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:26:28.572+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 33632 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:26:28.698 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:28.511+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:28.698 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:28.511+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:28.698 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:28.511+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:28.699 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:28.512+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:28.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:28.577+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:28.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:28.577+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:28.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:28.577+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:28.699 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:28.578+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:28.700 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:28.379+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:28.700 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:28.379+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:28.700 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:28.379+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:28.700 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:28.380+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:29.643 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:29.555+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:29.643 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:29.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:29.643 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:29.556+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:29.644 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:29.556+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:29.644 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:29.538+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:29.644 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:29.538+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:29.644 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:29.538+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:29.644 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:29.539+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:29.645 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:29.417+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:29.645 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:29.417+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:29.645 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:29.417+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:29.645 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:29.417+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:30.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:26:30.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:26:30.677 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:26:30.678 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T21:26:30.678 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-12-16T21:26:30.678 DEBUG:teuthology.orchestra.run.smithi052:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-16T21:26:30.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:26:30.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:26:30.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:30.553+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:30.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:30.553+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:30.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:30.553+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:30.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:30.554+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:30.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:30.446+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:30.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:30.446+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:30.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:30.446+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:30.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:30.446+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:30.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:30.561+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:30.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:30.561+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:30.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:30.561+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:30.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:30.562+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:31.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:31.517+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:31.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:31.517+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:31.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:31.517+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:31.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:31.518+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:31.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:31.513+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:31.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:31.513+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:31.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:31.513+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:31.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:31.514+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:31.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:31.464+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:31.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:31.464+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:31.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:31.464+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:31.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:31.464+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:32.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:32.506+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:32.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:32.506+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:32.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:32.506+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:32.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:32.506+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:32.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:32.487+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:32.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:32.487+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:32.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:32.487+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:32.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:32.488+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:32.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:32.537+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:32.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:32.537+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:32.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:32.537+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:32.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:32.538+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:33.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:26:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:26:33.578+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 33716 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:26:33.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:33.475+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:33.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:33.475+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:33.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:33.475+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:33.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:33.476+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:33.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:33.539+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:33.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:33.539+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:33.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:33.539+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:33.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:33.539+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:33.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:33.499+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:33.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:33.499+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:33.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:33.499+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:33.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:33.499+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:33.777 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:26:34.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:34.459+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:34.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:34.459+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:34.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:34.459+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:34.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:34.459+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:34.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:34.473+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:34.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:34.473+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:34.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:34.473+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:34.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:34.473+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:34.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:34.537+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:34.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:34.537+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:34.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:34.537+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:34.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:34.538+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:35.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:35.480+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:35.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:35.480+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:35.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:35.480+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:35.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:35.481+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:35.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:35.443+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:35.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:35.443+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:35.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:35.443+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:35.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:35.444+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:35.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:35.567+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:35.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:35.567+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:35.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:35.567+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:35.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:35.568+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:36.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:36.451+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:36.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:36.451+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:36.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:36.451+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:36.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:36.452+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:36.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:36.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:36.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:36.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:36.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:36.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:36.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:36.452+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:36.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:36.521+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:36.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:36.521+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:36.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:36.521+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:36.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:36.521+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:37.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:37.477+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:37.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:37.477+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:37.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:37.477+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:37.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:37.477+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:37.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:37.474+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:37.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:37.474+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:37.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:37.474+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:37.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:37.474+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:37.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:37.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:37.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:37.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:37.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:37.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:37.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:37.479+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:38.709 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:26:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:26:38.583+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 33802 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:26:38.710 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:38.454+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:38.710 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:38.454+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:38.710 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:38.454+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:38.710 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:38.455+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:38.711 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:38.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:38.711 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:38.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:38.711 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:38.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:38.711 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:38.440+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:38.711 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:38.467+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:38.711 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:38.467+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:38.711 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:38.467+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:38.712 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:38.467+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:39.676 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:39.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:39.677 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:39.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:39.677 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:39.451+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:39.677 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:39.452+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:39.677 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:39.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:39.678 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:39.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:39.678 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:39.479+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:39.678 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:39.479+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:39.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:39.405+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:39.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:39.405+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:39.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:39.405+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:39.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:39.406+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:40.519 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:40.448+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:40.519 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:40.448+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:40.520 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:40.448+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:40.520 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:40.448+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:40.520 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:40.493+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:40.520 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:40.493+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:40.520 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:40.493+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:40.520 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:40.493+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:40.780 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:40.517+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:40.781 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:40.517+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:40.781 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:40.517+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:40.781 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:40.517+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:41.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:41.530+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:41.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:41.530+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:41.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:41.530+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:41.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:41.530+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:41.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:41.456+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:41.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:41.456+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:41.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:41.456+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:41.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:41.457+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:41.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:41.514+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:41.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:41.514+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:41.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:41.514+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:41.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:41.515+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:42.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:42.485+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:42.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:42.485+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:42.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:42.485+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:42.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:42.485+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:42.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:42.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:42.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:42.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:42.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:42.563+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:42.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:42.564+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:42.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:42.427+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:42.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:42.427+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:42.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:42.427+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:42.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:42.427+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:42.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:26:42.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:26:43.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:26:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:26:43.589+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 33886 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:26:43.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:43.472+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:43.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:43.472+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:43.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:43.472+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:43.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:43.472+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:43.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:43.523+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:43.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:43.523+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:43.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:43.523+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:43.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:43.523+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:43.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:43.472+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:43.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:43.472+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:43.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:43.472+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:43.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:43.472+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:44.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:44.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:44.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:44.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:44.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:44.423+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:44.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:44.424+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:44.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:44.494+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:44.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:44.494+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:44.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:44.494+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:44.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:44.494+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:44.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:44.534+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:44.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:44.534+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:44.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:44.534+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:44.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:44.534+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:45.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:45.466+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:45.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:45.466+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:45.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:45.466+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:45.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:45.467+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:45.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:45.496+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:45.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:45.496+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:45.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:45.496+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:45.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:45.497+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:45.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:45.495+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:45.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:45.495+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:45.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:45.495+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:45.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:45.495+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:45.849 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:26:46.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:46.483+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:46.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:46.483+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:46.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:46.483+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:46.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:46.483+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:46.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:46.466+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:46.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:46.466+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:46.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:46.466+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:46.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:46.467+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:46.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:46.496+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:46.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:46.496+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:46.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:46.496+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:46.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:46.496+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:47.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:47.505+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:47.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:47.505+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:47.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:47.505+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:47.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:47.505+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:47.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:47.448+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:47.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:47.448+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:47.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:47.448+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:47.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:47.448+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:47.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:47.478+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:47.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:47.478+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:47.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:47.478+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:47.757 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:47.478+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:48.723 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:26:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:26:48.595+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 33972 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:26:48.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:48.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:48.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:48.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:48.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:48.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:48.724 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:48.485+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:48.725 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:48.435+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:48.725 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:48.435+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:48.725 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:48.435+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:48.725 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:48.435+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:48.725 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:48.505+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:48.725 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:48.505+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:48.726 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:48.505+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:48.726 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:48.505+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:49.750 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:49.518+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:49.750 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:49.518+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:49.750 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:49.518+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:49.750 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:49.518+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:49.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:49.480+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:49.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:49.480+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:49.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:49.480+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:49.751 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:49.480+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:49.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:49.421+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:49.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:49.421+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:49.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:49.421+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:49.752 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:49.421+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:50.757 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:50.558+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:50.758 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:50.558+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:50.758 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:50.558+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:50.758 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:50.558+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:50.759 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:50.494+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:50.759 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:50.494+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:50.759 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:50.494+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:50.759 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:50.494+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:50.760 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:50.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:50.760 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:50.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:50.760 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:50.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:50.760 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:50.406+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:51.580 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:51.466+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:51.580 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:51.466+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:51.580 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:51.466+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:51.581 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:51.466+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:51.581 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:51.363+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:51.581 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:51.363+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:51.581 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:51.363+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:51.581 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:51.364+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:51.578+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:52.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:51.578+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:52.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:51.578+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:52.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:51.578+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:52.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:52.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:52.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:52.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:52.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:52.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:52.464+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:52.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:52.412+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:52.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:52.412+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:52.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:52.412+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:52.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:52.412+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:52.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:52.604+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:52.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:52.604+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:52.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:52.604+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:52.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:52.605+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:53.569 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:53.383+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:53.570 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:53.383+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:53.570 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:53.383+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:53.570 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:53.383+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:53.570 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:53.492+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:53.571 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:53.492+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:53.571 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:53.492+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:53.571 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:53.492+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:54.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:26:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:26:53.600+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 34056 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:26:54.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:53.567+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:54.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:53.567+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:54.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:53.567+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:54.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:53.568+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:54.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:54.588+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:54.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:54.588+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:54.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:54.588+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:54.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:54.588+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:54.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:54.515+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:54.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:54.515+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:54.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:54.515+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:54.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:54.516+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:54.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:54.388+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:54.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:54.388+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:54.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:54.388+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:54.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:54.388+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:55.631 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:55.492+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:55.631 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:55.492+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:55.631 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:55.492+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:55.631 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:55.492+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:55.631 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:55.377+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:55.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:55.377+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:55.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:55.377+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:55.632 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:55.377+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:55.629+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:55.629+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:56.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:55.629+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:56.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:55.629+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:56.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:56.443+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:56.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:56.443+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:56.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:56.443+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:56.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:56.443+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:56.505 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:56.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:56.505 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:56.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:56.505 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:56.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:56.505 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:56.356+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:57.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:56.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:57.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:56.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:57.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:56.679+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:57.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:56.679+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:57.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:57.409+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:57.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:57.409+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:57.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:57.409+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:57.652 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:57.409+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:57.652 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:57.323+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:57.653 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:57.323+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:57.653 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:57.323+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:57.653 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:57.324+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:57.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:26:57.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:26:58.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:57.650+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:58.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:57.650+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:58.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:57.650+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:58.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:57.650+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:58.608 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:58.361+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:58.608 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:58.361+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:58.608 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:58.361+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:58.609 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:58.361+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:58.609 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:58.305+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:58.609 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:58.305+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:58.609 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:58.305+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:58.609 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:58.305+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:59.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:26:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:26:58.606+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 34142 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:26:59.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:58.682+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:59.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:58.682+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:59.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:58.682+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:26:59.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:58.682+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:59.718 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:59.291+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:59.718 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:59.291+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:59.718 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:59.291+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:26:59.719 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:26:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:26:59.291+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:26:59.719 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:59.366+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:59.719 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:59.366+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:59.719 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:59.366+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:26:59.719 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:26:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:26:59.366+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:00.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:59.716+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:00.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:59.716+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:00.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:59.716+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:00.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:26:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:26:59.716+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:00.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:00.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:00.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:00.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:00.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:00.412+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:00.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:00.412+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:00.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:00.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:00.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:00.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:00.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:00.246+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:00.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:00.246+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:00.914 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:27:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:00.689+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:01.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:00.689+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:01.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:00.689+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:01.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:00.689+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:01.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:01.415+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:01.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:01.415+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:01.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:01.415+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:01.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:01.415+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:01.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:01.238+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:01.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:01.238+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:01.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:01.238+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:01.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:01.238+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:02.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:01.686+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:02.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:01.686+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:02.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:01.686+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:02.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:01.686+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:02.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:02.415+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:02.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:02.415+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:02.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:02.415+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:02.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:02.415+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:02.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:02.236+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:02.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:02.236+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:02.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:02.236+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:02.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:02.237+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:03.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:02.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:03.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:02.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:03.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:02.637+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:03.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:02.637+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:03.614 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:03.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:03.614 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:03.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:03.614 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:03.428+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:03.614 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:03.428+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:03.614 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:03.285+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:03.615 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:03.285+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:03.615 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:03.285+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:03.615 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:03.285+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:03.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:27:03.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:27:03.985 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:27:03.986 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T21:27:03.986 INFO:tasks.cephadm:Archiving crash dumps... 2023-12-16T21:27:03.988 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494793/remote/smithi052/crash 2023-12-16T21:27:03.989 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/crash -- . 2023-12-16T21:27:04.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:27:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:27:03.612+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 34226 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:27:04.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:03.613+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:04.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:03.613+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:04.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:03.613+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:04.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:03.614+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:04.068 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494793/remote/smithi150/crash 2023-12-16T21:27:04.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:27:04.069 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:27:04.428 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:04.255+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:04.429 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:04.255+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:04.429 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:04.255+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:04.429 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:04.255+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:04.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:04.426+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:04.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:04.426+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:04.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:04.426+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:04.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:04.427+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:04.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:04.661+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:04.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:04.661+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:04.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:04.661+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:04.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:04.662+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:05.589 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:05.476+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:05.589 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:05.476+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:05.589 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:05.476+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:05.589 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:05.477+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:05.590 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:05.291+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:05.590 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:05.291+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:05.590 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:05.291+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:05.590 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:05.291+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:06.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:05.694+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:06.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:05.694+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:06.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:05.694+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:06.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:05.694+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:06.656 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:06.481+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:06.656 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:06.481+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:06.657 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:06.481+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:06.657 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:06.481+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:06.657 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:06.317+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:06.657 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:06.317+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:06.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:06.317+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:06.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:06.317+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:07.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:06.653+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:07.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:06.653+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:07.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:06.653+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:07.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:06.654+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:07.121 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:27:07.693 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:07.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:07.694 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:07.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:07.694 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:07.529+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:07.694 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:07.529+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:07.694 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:07.319+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:07.694 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:07.319+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:07.694 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:07.319+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:07.694 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:07.319+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:08.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:07.692+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:08.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:07.692+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:08.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:07.692+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:08.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:07.692+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:08.619 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:08.482+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:08.620 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:08.482+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:08.620 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:08.482+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:08.620 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:08.482+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:08.620 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:08.306+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:08.620 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:08.306+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:08.621 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:08.306+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:08.621 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:08.306+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:09.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:27:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:27:08.617+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 34312 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:27:09.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:08.689+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:09.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:08.689+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:09.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:08.689+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:09.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:08.689+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:09.583 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:09.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:09.583 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:09.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:09.583 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:09.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:09.584 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:09.454+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:09.584 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:09.337+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:09.584 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:09.337+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:09.584 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:09.337+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:09.584 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:09.337+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:10.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:09.677+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:10.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:09.677+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:10.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:09.677+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:10.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:09.677+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:10.633 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:10.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:10.633 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:10.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:10.634 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:10.454+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:10.634 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:10.455+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:10.634 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:10.319+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:10.634 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:10.319+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:10.634 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:10.319+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:10.634 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:10.319+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:11.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:10.631+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:11.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:10.631+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:11.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:10.631+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:11.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:10.631+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:11.634 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:11.502+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:11.635 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:11.502+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:11.635 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:11.502+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:11.635 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:11.502+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:11.635 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:11.365+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:11.635 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:11.365+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:11.635 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:11.365+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:11.636 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:11.366+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:11.633+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:12.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:11.633+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:12.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:11.633+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:12.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:11.633+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:12.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:12.415+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:12.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:12.415+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:12.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:12.415+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:12.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:12.415+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:12.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:12.510+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:12.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:12.510+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:12.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:12.510+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:12.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:12.511+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:12.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:12.608+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:12.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:12.608+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:12.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:12.608+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:12.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:12.609+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:13.750 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:27:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:27:13.623+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 34396 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:27:13.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:13.425+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:13.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:13.425+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:13.750 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:13.425+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:13.751 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:13.425+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:13.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:13.588+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:13.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:13.588+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:13.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:13.588+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:13.751 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:13.588+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:13.465+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:13.465+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:13.465+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:13.752 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:13.465+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:14.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:14.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:14.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:14.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:14.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:14.439+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:14.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:14.439+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:14.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:14.436+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:14.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:14.436+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:14.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:14.436+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:14.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:14.436+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:14.756 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:14.615+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:14.756 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:14.615+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:14.756 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:14.615+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:14.756 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:14.615+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:15.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:15.479+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:15.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:15.479+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:15.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:15.479+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:15.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:15.479+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:15.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:15.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:15.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:15.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:15.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:15.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:15.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:15.399+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:15.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:15.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:15.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:15.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:15.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:15.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:15.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:15.587+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:16.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:27:16.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:27:16.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:16.618+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:16.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:16.618+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:16.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:16.618+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:16.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:16.618+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:16.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:16.476+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:16.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:16.476+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:16.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:16.476+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:16.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:16.476+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:16.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:16.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:16.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:16.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:16.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:16.405+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:16.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:16.405+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:17.618 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:17.432+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:17.618 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:17.432+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:17.618 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:17.432+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:17.618 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:17.432+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:17.618 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:17.359+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:17.619 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:17.359+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:17.619 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:17.359+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:17.619 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:17.359+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:18.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:17.616+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:18.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:17.616+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:18.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:17.616+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:18.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:17.616+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:18.569 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:18.383+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:18.569 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:18.383+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:18.570 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:18.383+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:18.570 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:18.384+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:18.570 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:18.310+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:18.570 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:18.310+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:18.570 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:18.310+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:18.570 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:18.310+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:19.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:27:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:27:18.629+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 34481 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:27:19.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:18.568+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:19.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:18.568+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:19.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:18.568+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:19.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:18.568+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:19.189 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:27:19.579 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:19.407+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:19.579 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:19.407+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:19.579 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:19.407+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:19.579 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:19.407+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:19.579 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:19.276+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:19.579 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:19.276+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:19.580 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:19.276+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:19.580 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:19.276+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:20.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:19.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:20.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:19.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:20.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:19.577+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:20.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:19.577+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:20.560 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:20.361+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:20.560 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:20.361+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:20.560 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:20.361+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:20.560 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:20.362+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:20.561 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:20.304+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:20.561 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:20.304+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:20.561 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:20.304+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:20.561 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:20.304+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:21.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:20.557+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:21.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:20.557+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:21.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:20.558+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:21.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:20.559+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:21.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:21.513+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:21.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:21.513+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:21.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:21.513+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:21.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:21.514+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:21.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:21.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:21.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:21.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:21.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:21.339+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:21.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:21.339+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:21.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:21.300+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:21.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:21.300+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:21.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:21.300+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:21.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:21.300+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:22.356+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:22.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:22.356+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:22.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:22.356+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:22.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:22.357+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:22.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:22.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:22.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:22.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:22.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:22.540+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:22.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:22.540+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:22.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:22.304+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:22.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:22.304+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:22.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:22.304+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:22.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:22.304+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:23.637 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:23.345+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:23.637 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:23.345+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:23.638 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:23.345+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:23.638 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:23.346+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:23.638 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:23.544+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:23.638 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:23.544+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:23.638 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:23.544+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:23.638 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:23.545+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:23.639 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:23.316+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:23.639 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:23.316+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:23.639 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:23.316+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:23.639 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:23.316+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:24.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:27:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:27:23.634+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 34565 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:27:24.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:24.330+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:24.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:24.330+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:24.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:24.330+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:24.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:24.331+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:24.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:24.524+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:24.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:24.524+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:24.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:24.524+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:24.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:24.524+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:24.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:24.388+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:24.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:24.388+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:24.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:24.388+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:24.756 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:24.388+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:25.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:25.339+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:25.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:25.339+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:25.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:25.339+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:25.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:25.339+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:25.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:25.495+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:25.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:25.495+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:25.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:25.495+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:25.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:25.495+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:25.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:25.398+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:25.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:25.398+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:25.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:25.398+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:25.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:25.398+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:26.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:26.312+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:26.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:26.312+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:26.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:26.312+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:26.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:26.312+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:26.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:26.509+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:26.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:26.509+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:26.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:26.509+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:26.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:26.509+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:26.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:26.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:26.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:26.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:26.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:26.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:26.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:26.440+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:27.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:27.360+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:27.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:27.360+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:27.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:27.360+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:27.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:27.360+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:27.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:27.465+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:27.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:27.465+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:27.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:27.465+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:27.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:27.465+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:27.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:27.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:27.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:27.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:27.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:27.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:27.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:27.430+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:28.643 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:28.514+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:28.644 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:28.514+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:28.644 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:28.514+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:28.644 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:28.514+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:28.644 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:28.465+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:28.644 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:28.465+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:28.644 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:28.465+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:28.645 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:28.465+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:28.645 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:28.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:28.645 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:28.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:28.645 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:28.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:28.645 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:28.369+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:29.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:27:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:27:28.640+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 34650 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:27:29.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:29.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:29.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:29.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:29.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:29.369+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:29.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:29.369+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:29.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:29.484+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:29.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:29.484+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:29.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:29.484+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:29.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:29.484+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:29.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:29.502+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:29.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:29.502+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:29.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:29.502+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:29.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:29.502+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:30.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:30.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:30.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:30.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:30.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:30.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:30.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:30.374+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:30.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:30.517+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:30.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:30.517+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:30.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:30.517+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:30.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:30.517+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:30.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:30.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:30.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:30.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:30.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:30.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:30.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:30.526+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:31.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:27:31.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:27:31.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:31.500+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:31.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:31.500+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:31.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:31.500+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:31.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:31.500+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:31.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:31.523+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:31.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:31.523+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:31.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:31.523+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:31.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:31.523+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:31.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:31.393+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:31.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:31.393+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:31.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:31.393+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:31.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:31.393+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:32.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:32.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:32.526+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:32.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:32.526+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:32.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:32.417+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:32.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:32.417+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:32.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:32.418+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:32.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:32.418+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:32.756 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:32.556+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:32.756 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:32.556+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:32.756 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:32.556+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:32.756 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:32.556+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:33.503 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:33.479+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:33.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:33.479+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:33.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:33.479+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:33.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:33.479+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:33.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:33.422+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:33.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:33.422+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:33.505 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:33.422+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:33.505 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:33.422+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:34.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:27:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:27:33.646+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 34734 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:27:34.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:33.596+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:34.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:33.596+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:34.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:33.596+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:34.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:33.596+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:34.257 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:27:34.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:34.599+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:34.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:34.599+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:34.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:34.599+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:34.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:34.599+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:34.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:34.444+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:34.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:34.444+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:34.756 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:34.444+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:34.756 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:34.444+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:34.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:34.469+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:34.757 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:34.469+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:34.757 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:34.469+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:34.757 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:34.469+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:35.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:35.640+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:35.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:35.640+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:35.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:35.640+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:35.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:35.640+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:35.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:35.478+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:35.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:35.478+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:35.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:35.478+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:35.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:35.479+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:35.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:35.437+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:35.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:35.437+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:35.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:35.437+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:35.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:35.437+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:36.596 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:36.444+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:36.597 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:36.444+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:36.597 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:36.444+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:36.597 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:36.444+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:36.597 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:36.438+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:36.597 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:36.438+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:36.598 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:36.438+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:36.598 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:36.438+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:37.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:36.593+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:37.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:36.593+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:37.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:36.593+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:37.004 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:36.594+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:37.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:27:37.259 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:27:37.329 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:27:37.330 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T21:27:37.330 INFO:tasks.cephadm:Checking cluster log for badness... 2023-12-16T21:27:37.330 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-16T21:27:37.373 INFO:teuthology.orchestra.run.smithi052.stderr:grep: /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph.log: No such file or directory 2023-12-16T21:27:37.374 INFO:tasks.cephadm:Compressing logs... 2023-12-16T21:27:37.374 DEBUG:teuthology.orchestra.run.smithi052:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-16T21:27:37.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:27:37.418 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:27:37.448 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-16T21:27:37.449 INFO:teuthology.orchestra.run.smithi052.stderr:: No such file or directory 2023-12-16T21:27:37.449 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi052.bvmosa.log 2023-12-16T21:27:37.449 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-mgr.smithi052.nmkcbs.log 2023-12-16T21:27:37.450 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-osd.1.log 2023-12-16T21:27:37.450 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi052.bvmosa.log: /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-mgr.smithi052.nmkcbs.log: gzip -5 --verbose -- /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-mon.smithi052.log 2023-12-16T21:27:37.450 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-osd.0.log 2023-12-16T21:27:37.452 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-osd.1.log: /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-mon.smithi052.log: gzip -5 --verbose -- /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-osd.2.log 2023-12-16T21:27:37.452 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-volume.log 2023-12-16T21:27:37.452 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi052.szlsle.log 2023-12-16T21:27:37.458 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-volume.log: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-16T21:27:37.459 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi052.szlsle.log: 90.8% -- replaced with /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-volume.log.gz 2023-12-16T21:27:37.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:37.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:37.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:37.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:37.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:37.587+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:37.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:37.587+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:37.756 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:37.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:37.756 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:37.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:37.756 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:37.485+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:37.756 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:37.485+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:37.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:37.423+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:37.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:37.423+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:37.757 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:37.423+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:37.757 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:37 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:37.424+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:38.058 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-mgr.smithi052.nmkcbs.log: file size changed while zipping 2023-12-16T21:27:38.059 INFO:teuthology.orchestra.run.smithi052.stderr: 89.9% -- replaced with /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-mgr.smithi052.nmkcbs.log.gz 2023-12-16T21:27:38.471 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:38.468+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:38.601 INFO:teuthology.orchestra.run.smithi052.stderr: 95.5% -- replaced with /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi052.bvmosa.log.gz 2023-12-16T21:27:38.737 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-osd.2.log: file size changed while zipping 2023-12-16T21:27:38.737 INFO:teuthology.orchestra.run.smithi052.stderr: 93.1% -- replaced with /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-osd.2.log.gz 2023-12-16T21:27:38.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:27:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:27:38.650+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 34820 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:27:38.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:38.468+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:38.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:38.468+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:38.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:38.469+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:38.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:38.496+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:38.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:38.496+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:38.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:38.496+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:38.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:38.496+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:38.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:38.592+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:38.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:38.592+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:38.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:38.592+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:38.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:38 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:38.593+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:38.851 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-mon.smithi052.log: file size changed while zipping 2023-12-16T21:27:38.852 INFO:teuthology.orchestra.run.smithi052.stderr: 92.8% -- replaced with /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-mon.smithi052.log.gz 2023-12-16T21:27:39.160 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-osd.0.log: file size changed while zipping 2023-12-16T21:27:39.161 INFO:teuthology.orchestra.run.smithi052.stderr: 90.8% -- replaced with /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-osd.0.log.gz 2023-12-16T21:27:39.181 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-osd.1.log: file size changed while zipping 2023-12-16T21:27:39.181 INFO:teuthology.orchestra.run.smithi052.stderr: 92.3% -- replaced with /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-osd.1.log.gz 2023-12-16T21:27:39.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:39.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:39.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:39.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:39.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:39.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:39.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:39.433+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:39.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:39.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:39.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:39.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:39.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:39.464+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:39.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:39.464+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:39.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:39.559+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:39.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:39.559+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:39.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:39.559+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:39.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:39 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:39.560+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:40.497 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:27:40.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:40.511+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:40.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:40.511+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:40.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:40.511+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:40.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:40.511+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:40.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:40.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:40.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:40.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:40.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:40.396+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:40.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:40.396+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:40.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:40.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:40.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:40.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:40.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:40.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:40.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:40 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:40.424+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:41.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:41.560+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:41.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:41.560+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:41.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:41.560+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:41.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:41.561+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:41.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:41.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:41.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:41.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:41.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:41.356+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:41.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:41.356+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:41.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:41.399+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:41.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:41.399+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:41.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:41.399+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:41.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:41 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:41.400+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:42.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:42.559+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:42.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:42.559+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:42.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:42.559+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:42.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:42.560+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:42.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:42.373+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:42.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:42.373+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:42.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:42.373+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:42.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:42.373+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:42.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:42.432+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:42.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:42.432+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:42.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:42.432+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:42.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:42 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:42.433+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:43.657 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:43.387+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:43.657 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:43.387+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:43.657 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:43.387+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:43.658 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:43.387+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:43.658 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:43.460+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:43.658 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:43.460+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:43.658 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:43.460+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:43.658 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:43.460+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:43.659 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:43.537+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:43.659 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:43.537+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:43.659 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:43.537+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:43.659 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:43.537+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:44.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:27:43 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:27:43.655+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 34904 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:27:44.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:44.503+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:44.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:44.503+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:44.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:44.503+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:44.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:44.503+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:44.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:44.421+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:44.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:44.421+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:44.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:44.421+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:44.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:44.421+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:44.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:44.364+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:44.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:44.364+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:44.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:44.364+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:44.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:44 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:44.364+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:45.747 INFO:teuthology.orchestra.run.smithi052.stderr: 92.7% -- replaced with /var/log/ceph/cf043a8a-9c53-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi052.szlsle.log.gz 2023-12-16T21:27:45.748 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:45.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:45.748 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:45.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:45.748 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:45.453+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:45.748 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:45.453+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:45.748 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:45.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:45.749 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:45.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:45.749 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:45.408+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:45.749 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:45.408+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:45.749 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:45.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:45.749 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:45.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:45.750 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:45.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:45.750 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:45 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:45.516+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:45.750 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-12-16T21:27:45.751 INFO:teuthology.orchestra.run.smithi052.stderr:real 0m8.318s 2023-12-16T21:27:45.751 INFO:teuthology.orchestra.run.smithi052.stderr:user 0m15.759s 2023-12-16T21:27:45.751 INFO:teuthology.orchestra.run.smithi052.stderr:sys 0m0.428s 2023-12-16T21:27:46.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:46.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:46.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:46.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:46.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:46.462+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:46.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:46.462+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:46.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:46.501+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:46.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:46.501+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:46.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:46.501+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:46.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:46.501+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:46.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:46.384+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:46.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:46.384+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:46.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:46.384+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:46.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:46 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:46.384+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:47.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:47.460+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:47.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:47.460+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:47.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:47.461+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:47.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:47.461+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:47.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:47.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:47.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:47.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:47.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:47.433+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:47.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:47.434+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:47.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:47.381+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:47.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:47.381+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:47.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:47.381+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:47.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:47 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:47.381+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:48.663 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:48.396+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:48.663 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:48.396+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:48.663 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:48.396+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:48.664 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:48.396+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:48.664 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:48.455+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:48.664 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:48.455+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:48.664 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:48.455+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:48.664 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:48.456+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:48.664 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:48.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:48.665 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:48.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:48.665 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:48.399+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:48.665 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:48.399+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:49.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:27:48 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:27:48.661+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 34990 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:27:49.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:27:49.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:27:49.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:49.465+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:49.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:49.465+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:49.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:49.465+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:49.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:49.466+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:49.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:49.368+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:49.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:49.368+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:49.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:49.368+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:49.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:49.368+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:49.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:49.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:49.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:49.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:49.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:49.416+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:49.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:49 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:49.416+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:50.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:50.475+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:50.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:50.475+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:50.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:50.475+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:50.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:50.475+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:50.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:50.361+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:50.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:50.361+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:50.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:50.361+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:50.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:50.361+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:50.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:50.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:50.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:50.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:50.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:50.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:50.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:50 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:50.466+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:51.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:51.450+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:51.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:51.450+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:51.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:51.450+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:51.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:51.450+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:51.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:51.384+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:51.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:51.384+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:51.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:51.384+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:51.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:51.384+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:51.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:51.428+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:51.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:51.428+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:51.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:51.428+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:51.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:51 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:51.428+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:52.565 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:27:52.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:52.407+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:52.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:52.407+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:52.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:52.407+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:52.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:52.407+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:52.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:52.422+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:52.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:52.422+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:52.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:52.422+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:52.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:52.423+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:52.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:52.430+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:52.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:52.430+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:52.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:52.430+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:52.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:52 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:52.431+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:53.669 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:53.378+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:53.669 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:53.378+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:53.669 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:53.378+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:53.670 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:53.378+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:53.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:53.447+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:53.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:53.447+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:53.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:53.447+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:53.670 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:53.448+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:53.670 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:53.412+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:53.671 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:53.412+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:53.671 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:53.412+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:53.671 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:53.413+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:54.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:27:53 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:27:53.667+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 35074 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:27:54.741 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:54.482+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:54.741 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:54.482+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:54.742 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:54.482+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:54.742 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:54.483+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:54.742 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:54.389+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:54.742 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:54.389+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:54.742 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:54.389+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:54.743 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:54.390+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:54.743 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:54.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:54.743 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:54.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:54.743 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:54.374+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:54.743 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:54 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:54.375+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:55.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:55.421+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:55.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:55.421+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:55.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:55.421+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:55.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:55.422+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:55.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:55.492+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:55.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:55.492+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:55.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:55.492+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:55.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:55.493+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:55.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:55.363+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:55.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:55.363+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:55.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:55.363+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:55.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:55 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:55.364+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:56.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:56.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:56.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:56.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:56.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:56.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:56.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:56.517+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:56.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:56.322+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:56.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:56.322+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:56.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:56.322+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:56.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:56.322+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:56.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:56.375+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:56.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:56.375+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:56.756 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:56.375+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:56.756 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:56 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:56.376+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:57.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:57.375+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:57.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:57.375+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:57.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:57.375+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:57.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:57.375+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:57.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:57.489+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:57.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:57.489+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:57.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:57.489+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:57.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:57.489+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:57.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:57.349+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:57.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:57.349+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:57.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:57.349+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:57.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:57 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:57.349+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:58.675 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:58.328+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:58.675 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:58.328+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:58.676 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:58.328+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:58.676 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:58.328+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:58.676 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:58.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:58.676 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:58.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:58.676 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:58.307+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:58.676 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:58.307+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:58.677 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:58.531+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:58.677 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:58.531+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:58.677 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:58.531+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:58.677 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:58.532+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:59.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:27:58 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:27:58.673+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 35159 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:27:59.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:59.376+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:59.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:59.376+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:59.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:59.376+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:27:59.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:27:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:27:59.376+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:59.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:59.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:59.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:59.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:59.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:59.516+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:27:59.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:27:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:27:59.516+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:27:59.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:59.353+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:59.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:59.353+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:59.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:59.353+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:27:59.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:27:59 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:27:59.353+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:00.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:00.381+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:00.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:00.381+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:00.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:00.381+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:00.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:00.381+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:00.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:00.492+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:00.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:00.492+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:00.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:00.492+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:00.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:00.493+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:00.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:00.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:00.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:00.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:00.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:00.398+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:00.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:00 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:00.398+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:01.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:01.338+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:01.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:01.338+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:01.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:01.338+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:01.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:01.338+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:01.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:01.464+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:01.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:01.464+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:01.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:01.464+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:01.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:01.464+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:01.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:01.355+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:01.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:01.355+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:01.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:01.355+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:01.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:01 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:01.355+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:02.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:02.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:02.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:02.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:02.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:02.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:02.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:02.374+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:02.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:02.460+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:02.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:02.460+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:02.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:02.460+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:02.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:02.461+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:02.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:02.386+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:02.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:02.386+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:02.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:02.386+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:02.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:02 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:02.386+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:03.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:03.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:03.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:03.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:03.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:03.423+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:03.678 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:03.423+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:03.679 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:03.415+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:03.679 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:03.415+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:03.679 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:03.415+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:03.679 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:03.415+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:03.679 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:03.503+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:03.679 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:03.503+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:03.679 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:03.503+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:03.680 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:03.503+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:03.957 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:28:03 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:28:03.679+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 35244 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:28:04.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:28:04.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:28:04.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:04.503+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:04.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:04.503+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:04.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:04.503+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:04.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:04.503+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:04.757 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:04.444+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:04.757 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:04.444+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:04.758 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:04.444+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:04.758 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:04.444+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:04.758 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:04.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:04.758 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:04.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:04.758 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:04.432+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:04.758 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:04 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:04.432+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:05.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:05.480+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:05.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:05.480+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:05.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:05.480+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:05.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:05.481+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:05.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:05.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:05.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:05.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:05.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:05.430+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:05.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:05.430+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:05.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:05.473+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:05.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:05.473+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:05.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:05.473+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:05.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:05 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:05.473+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:06.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:06.468+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:06.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:06.468+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:06.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:06.468+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:06.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:06.469+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:06.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:06.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:06.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:06.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:06.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:06.465+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:06.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:06.465+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:06.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:06.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:06.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:06.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:06.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:06.452+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:06.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:06 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:06.452+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:07.637 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:28:07.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:07.493+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:07.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:07.493+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:07.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:07.493+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:07.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:07.493+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:07.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:07.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:07.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:07.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:07.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:07.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:07.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:07.440+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:07.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:07.510+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:07.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:07.510+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:07.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:07.510+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:07.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:07 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:07.510+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:08.687 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:08.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:08.688 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:08.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:08.688 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:08.403+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:08.688 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:08.404+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:08.688 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:08.539+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:08.688 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:08.539+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:08.688 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:08.539+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:08.689 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:08.540+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:08.689 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:08.529+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:08.689 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:08.529+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:08.689 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:08.529+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:08.689 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:08.530+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:09.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:28:08 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:28:08.685+0000 7f671caf5700 -1 mon.smithi052@0(probing) e2 get_health_metrics reporting 35329 slow ops, oldest is log(1 entries from seq 228 at 2023-12-16T20:50:57.939711+0000) 2023-12-16T21:28:09.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:09.382+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:09.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:09.382+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:09.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:09.382+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:09.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:09.382+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:09.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:09.548+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:09.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:09.548+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:09.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:09.548+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:09.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:09.548+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:09.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:09.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:09.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:09.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:09.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:09.520+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:09.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:09 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:09.520+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:10.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:28:10.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:28:10.705 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:28:10.706 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T21:28:10.706 INFO:tasks.cephadm:Removing cluster... 2023-12-16T21:28:10.706 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 --force 2023-12-16T21:28:10.740 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:10.530+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:10.740 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:10.530+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:10.740 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:10.530+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:10.740 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:10.530+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:10.741 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:10.336+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:10.741 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:10.336+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:10.741 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:10.336+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:10.741 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:10.336+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:10.742 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:10.532+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:10.742 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:10.532+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:10.742 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:10.532+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:10.742 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:10 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:10.533+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:11.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:28:10 smithi052 systemd[1]: Stopping Ceph mon.smithi052 for cf043a8a-9c53-11ee-95a5-87774f69a715... 2023-12-16T21:28:11.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:28:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:28:11.196+0000 7f671fafb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi052 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-12-16T21:28:11.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:28:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052[80757]: 2023-12-16T21:28:11.196+0000 7f671fafb700 -1 mon.smithi052@0(probing) e2 *** Got Signal Terminated *** 2023-12-16T21:28:11.655 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:28:11 smithi052 podman[138028]: 2023-12-16 21:28:11.54230125 +0000 UTC m=+0.406538910 container died 094cffecdaf4e42894642076a08e43bf1d5979bb934b4a17e295705b21293ce1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=quincy-1f2f551, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2023-12-16T21:28:11.655 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:11.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:11.656 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:11.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:11.656 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:11.374+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:11.656 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:11.374+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:11.656 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:11.499+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:11.656 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:11.499+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:11.656 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:11.499+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:11.657 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:11.499+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:11.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:11.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:11.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:11.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:11.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:11.576+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:11.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:11 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:11.576+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:12.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:28:11 smithi052 podman[138028]: 2023-12-16 21:28:11.710478862 +0000 UTC m=+0.574716516 container cleanup 094cffecdaf4e42894642076a08e43bf1d5979bb934b4a17e295705b21293ce1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-1f2f551, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.31.3, org.label-schema.build-date=20231212) 2023-12-16T21:28:12.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:28:11 smithi052 bash[138028]: ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052 2023-12-16T21:28:12.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:28:11 smithi052 podman[138042]: 2023-12-16 21:28:11.876904197 +0000 UTC m=+0.335612983 container remove 094cffecdaf4e42894642076a08e43bf1d5979bb934b4a17e295705b21293ce1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-mon-smithi052, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-1f2f551, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2023-12-16T21:28:12.392 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:28:12 smithi052 systemd[1]: ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@mon.smithi052.service: Deactivated successfully. 2023-12-16T21:28:12.393 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:28:12 smithi052 systemd[1]: Stopped Ceph mon.smithi052 for cf043a8a-9c53-11ee-95a5-87774f69a715. 2023-12-16T21:28:12.393 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Dec 16 21:28:12 smithi052 systemd[1]: ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@mon.smithi052.service: Consumed 28.769s CPU time. 2023-12-16T21:28:12.736 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:12.562+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:12.736 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:12.562+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:12.737 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:12.562+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:12.737 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:12.562+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:12.737 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:12.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:12.737 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:12.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:12.737 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:12.390+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:12.738 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:12.390+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:12.738 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:12.526+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:12.738 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:12.526+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:12.738 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:12.526+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:12.738 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:12 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:12.526+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:13.480 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:13.371+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:13.480 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:13.371+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:13.480 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:13.371+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:13.480 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:13.372+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:13.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:13.483+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:13.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:13.483+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:13.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:13.483+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:13.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:13.483+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:13.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:13.603+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:13.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:13.603+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:13.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:13.603+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:13.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:13 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:13.604+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:14.456 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:14.355+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:14.456 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:14.355+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:14.456 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:14.355+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:14.456 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:14.355+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:14.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:14.593+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:14.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:14.593+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:14.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:14.594+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:14.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:14.594+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:14.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:14.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:14.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:14.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:14.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:14.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:14.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:14 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:14.454+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:15.352 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:15.344+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:15.353 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:15.344+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:15.353 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:15.344+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:15.353 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:15.345+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:15.635 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:15.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:15.636 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:15.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:15.636 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:15.454+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:15.636 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:15.454+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:15.890 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:15.633+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:15.891 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:15.633+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:15.891 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:15.633+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:15.891 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:15 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:15.634+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:16.430 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:16.313+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:16.431 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:16.313+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:16.431 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:16.313+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:16.431 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:16.313+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:16.682 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:16.601+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:16.682 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:16.601+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:16.682 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:16.601+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:16.682 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:16.602+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:16.683 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:16.428+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:16.683 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:16.428+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:16.683 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:16.428+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:16.683 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:16 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:16.429+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:17.568 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:17.334+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:17.569 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:17.334+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:17.569 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:17.334+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:17.569 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:17.335+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:17.569 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:17.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:17.569 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:17.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:17.570 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:17.474+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:17.570 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:17.474+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:17.570 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:17 smithi052 systemd[1]: Stopping Ceph osd.0 for cf043a8a-9c53-11ee-95a5-87774f69a715... 2023-12-16T21:28:17.570 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:17.469+0000 7f9c5de82700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-12-16T21:28:17.570 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:17.469+0000 7f9c5de82700 -1 osd.0 43 *** Got signal Terminated *** 2023-12-16T21:28:17.570 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:17.469+0000 7f9c5de82700 -1 osd.0 43 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T21:28:17.570 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:17.566+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:17.571 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:17.566+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:17.571 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:17.566+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:18.003 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:17 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:17.567+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:18.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:18.327+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:18.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:18.327+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:18.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:18.327+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:18.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:18.328+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:18.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:18.480+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:18.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:18.480+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:18.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:18.480+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:18.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:18.480+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:18.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:18.525+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:18.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:18.525+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:18.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:18.525+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:18.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:18 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:18.525+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:19.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:19.519+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:19.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:19.519+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:19.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:19.519+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:19.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:19.519+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:19.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:19.378+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:19.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:19.378+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:19.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:19.378+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:19.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:19.378+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:19.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:19.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:19.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:19.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:19.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:19.466+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:19.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:19 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:19.466+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:20.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:20.406+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:20.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:20.406+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:20.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:20.406+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:20.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:20.406+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:20.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:20.564+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:20.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:20.564+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:20.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:20.564+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:20.755 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:20.564+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:20.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:20.429+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:20.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:20.429+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:20.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:20.429+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:20.756 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:20 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:20.429+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:21.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:21.558+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:21.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:21.558+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:21.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:21.558+0000 7f9c5847c700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:34.972166+0000 (oldest deadline 2023-12-16T21:22:54.972166+0000) 2023-12-16T21:28:21.754 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0[99291]: 2023-12-16T21:28:21.558+0000 7f9c5847c700 -1 osd.0 43 get_health_metrics reporting 112 slow ops, oldest is osd_op(client.24307.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:21.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:21.419+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:21.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:21.419+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:21.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:21.419+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:21.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:21.419+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:21.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:21.456+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:21.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:21.456+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:21.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:21.456+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:21.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:21 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:21.456+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:22.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:22.467+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:22.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:22.467+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:22.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:22.467+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:22.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:22.468+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:22.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:22.388+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:22.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:22.388+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:22.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:22.388+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:22.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:22 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:22.388+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:23.253 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:22 smithi052 podman[138872]: 2023-12-16 21:28:22.974986379 +0000 UTC m=+5.566194347 container died 70d69be336b3fb6fb42ea6c17555baee391d11411c0b19bf89b7e1538e190f27 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, RELEASE=quincy-1f2f551, ceph=True, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2023-12-16T21:28:23.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:23 smithi052 podman[138872]: 2023-12-16 21:28:23.159718999 +0000 UTC m=+5.750926966 container cleanup 70d69be336b3fb6fb42ea6c17555baee391d11411c0b19bf89b7e1538e190f27 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-1f2f551, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.31.3) 2023-12-16T21:28:23.254 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:23 smithi052 bash[138872]: ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0 2023-12-16T21:28:23.753 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:23 smithi052 podman[138890]: 2023-12-16 21:28:23.367124739 +0000 UTC m=+0.388674106 container remove 70d69be336b3fb6fb42ea6c17555baee391d11411c0b19bf89b7e1538e190f27 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0, RELEASE=quincy-1f2f551, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.31.3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2023-12-16T21:28:23.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:23.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:23.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:23.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:23.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:23.440+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:23.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:23.440+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:23.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:23.393+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:23.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:23.393+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:23.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:23.393+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:23.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:23 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:23.394+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:24.256 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:23 smithi052 podman[138980]: 2023-12-16 21:28:23.687069114 +0000 UTC m=+0.017920664 image pull 60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5 quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112 2023-12-16T21:28:24.257 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:23 smithi052 podman[138980]: 2023-12-16 21:28:23.85037745 +0000 UTC m=+0.181229042 container create a8d9ae1ad0fbb172df5be87a26f799b936522128f5d03a2c5dc370553a82f710 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=quincy-1f2f551, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20231212, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.31.3) 2023-12-16T21:28:24.257 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:23 smithi052 podman[138980]: 2023-12-16 21:28:23.959537568 +0000 UTC m=+0.290389135 container init a8d9ae1ad0fbb172df5be87a26f799b936522128f5d03a2c5dc370553a82f710 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0-deactivate, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-1f2f551, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2023-12-16T21:28:24.257 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:23 smithi052 podman[138980]: 2023-12-16 21:28:23.968704105 +0000 UTC m=+0.299555655 container start a8d9ae1ad0fbb172df5be87a26f799b936522128f5d03a2c5dc370553a82f710 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.31.3, RELEASE=quincy-1f2f551, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2023-12-16T21:28:24.257 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:24 smithi052 podman[138980]: 2023-12-16 21:28:24.000306454 +0000 UTC m=+0.331158025 container attach a8d9ae1ad0fbb172df5be87a26f799b936522128f5d03a2c5dc370553a82f710 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, RELEASE=quincy-1f2f551, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2023-12-16T21:28:24.526 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:24 smithi052 podman[138980]: 2023-12-16 21:28:24.265347158 +0000 UTC m=+0.596198709 container died a8d9ae1ad0fbb172df5be87a26f799b936522128f5d03a2c5dc370553a82f710 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, RELEASE=quincy-1f2f551, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2023-12-16T21:28:24.527 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:24.479+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:24.527 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:24.479+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:24.527 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:24.479+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:24.527 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:24.480+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:24.528 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:24.413+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:24.528 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:24.413+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:24.528 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:24.413+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:24.528 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:24.413+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:24.971 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:24 smithi052 systemd[1]: Stopping Ceph osd.1 for cf043a8a-9c53-11ee-95a5-87774f69a715... 2023-12-16T21:28:24.971 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:24.968+0000 7fe36793d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-12-16T21:28:24.971 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:24.968+0000 7fe36793d700 -1 osd.1 43 *** Got signal Terminated *** 2023-12-16T21:28:24.971 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:24 smithi052 podman[139007]: 2023-12-16 21:28:24.525205354 +0000 UTC m=+0.250521554 container remove a8d9ae1ad0fbb172df5be87a26f799b936522128f5d03a2c5dc370553a82f710 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-0-deactivate, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.31.3, RELEASE=quincy-1f2f551, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2023-12-16T21:28:24.972 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:24 smithi052 systemd[1]: ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@osd.0.service: Deactivated successfully. 2023-12-16T21:28:24.972 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:24 smithi052 systemd[1]: Stopped Ceph osd.0 for cf043a8a-9c53-11ee-95a5-87774f69a715. 2023-12-16T21:28:24.972 INFO:journalctl@ceph.osd.0.smithi052.stdout:Dec 16 21:28:24 smithi052 systemd[1]: ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@osd.0.service: Consumed 20.639s CPU time. 2023-12-16T21:28:25.253 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:24 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:24.968+0000 7fe36793d700 -1 osd.1 43 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T21:28:25.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:25.514+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:25.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:25.514+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:25.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:25.514+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:25.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:25.514+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:25.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:25.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:25.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:25.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:25.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:25.389+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:25.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:25 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:25.389+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:26.541+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:26.541+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:26.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:26.541+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:26.755 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:26.541+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:26.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:26.358+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:26.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:26.358+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:26.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:26.358+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:26.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:26 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:26.358+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:27.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:27.520+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:27.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:27.520+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:27.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:27.520+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:27.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:27.520+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:27.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:27.395+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:27.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:27.395+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:27.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:27.395+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:27.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:27 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:27.396+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:28.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:28.541+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:28.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:28.541+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:28.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:28.541+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:28.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:28.542+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:28.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:28.400+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:28.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:28.400+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:28.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:28.400+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:28.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:28 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:28.400+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:29.753 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:29.502+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:29.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:29.502+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:29.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:29.502+0000 7fe362f39700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:44.169300+0000 (oldest deadline 2023-12-16T21:23:04.169300+0000) 2023-12-16T21:28:29.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1[106744]: 2023-12-16T21:28:29.502+0000 7fe362f39700 -1 osd.1 43 get_health_metrics reporting 89 slow ops, oldest is osd_op(client.24307.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:29.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:29.415+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:29.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:29.415+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:29.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:29.415+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:29.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:29 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:29.415+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:30.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:30 smithi052 podman[139094]: 2023-12-16 21:28:30.482944466 +0000 UTC m=+5.583525706 container died a39f1afa729547efa50212ab452c5b8ce2272b0ae4c2d376a2ad47de2ea3a125 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, RELEASE=quincy-1f2f551, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2023-12-16T21:28:30.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:30 smithi052 podman[139094]: 2023-12-16 21:28:30.632942805 +0000 UTC m=+5.733524025 container cleanup a39f1afa729547efa50212ab452c5b8ce2272b0ae4c2d376a2ad47de2ea3a125 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, RELEASE=quincy-1f2f551, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2023-12-16T21:28:30.754 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:30 smithi052 bash[139094]: ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1 2023-12-16T21:28:30.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:30.402+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:30.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:30.402+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:30.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:30.402+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:30.755 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:30 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:30.402+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:31.167 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:30 smithi052 podman[139116]: 2023-12-16 21:28:30.766030763 +0000 UTC m=+0.278392204 container remove a39f1afa729547efa50212ab452c5b8ce2272b0ae4c2d376a2ad47de2ea3a125 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, io.buildah.version=1.31.3, RELEASE=quincy-1f2f551, CEPH_POINT_RELEASE=, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2023-12-16T21:28:31.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:31.385+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:31.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:31.385+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:31.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:31.385+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:31.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:31 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:31.385+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:31.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:31 smithi052 podman[139206]: 2023-12-16 21:28:31.166031421 +0000 UTC m=+0.064402335 container create c7b4463d424e2180a79c4d116c65f94fde2d715ddd82d24f175848ab6916f8d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.31.3, GIT_CLEAN=True, RELEASE=quincy-1f2f551) 2023-12-16T21:28:31.504 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:31 smithi052 podman[139206]: 2023-12-16 21:28:31.119799138 +0000 UTC m=+0.018170064 image pull 60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5 quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112 2023-12-16T21:28:31.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:31 smithi052 podman[139206]: 2023-12-16 21:28:31.258207002 +0000 UTC m=+0.156577925 container init c7b4463d424e2180a79c4d116c65f94fde2d715ddd82d24f175848ab6916f8d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.31.3, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=quincy-1f2f551, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux ) 2023-12-16T21:28:31.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:31 smithi052 podman[139206]: 2023-12-16 21:28:31.264999373 +0000 UTC m=+0.163370287 container start c7b4463d424e2180a79c4d116c65f94fde2d715ddd82d24f175848ab6916f8d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.31.3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=quincy-1f2f551, org.label-schema.build-date=20231212) 2023-12-16T21:28:31.505 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:31 smithi052 podman[139206]: 2023-12-16 21:28:31.299342218 +0000 UTC m=+0.197713151 container attach c7b4463d424e2180a79c4d116c65f94fde2d715ddd82d24f175848ab6916f8d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1-deactivate, io.buildah.version=1.31.3, org.label-schema.schema-version=1.0, RELEASE=quincy-1f2f551, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20231212, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2023-12-16T21:28:31.844 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:31 smithi052 podman[139206]: 2023-12-16 21:28:31.574188602 +0000 UTC m=+0.472559538 container died c7b4463d424e2180a79c4d116c65f94fde2d715ddd82d24f175848ab6916f8d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-1f2f551, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.31.3, org.label-schema.build-date=20231212, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2023-12-16T21:28:31.844 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:31 smithi052 podman[139232]: 2023-12-16 21:28:31.816012952 +0000 UTC m=+0.237016537 container remove c7b4463d424e2180a79c4d116c65f94fde2d715ddd82d24f175848ab6916f8d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-1-deactivate, io.buildah.version=1.31.3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=quincy-1f2f551, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2023-12-16T21:28:31.844 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:31 smithi052 systemd[1]: ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@osd.1.service: Deactivated successfully. 2023-12-16T21:28:31.844 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:31 smithi052 systemd[1]: Stopped Ceph osd.1 for cf043a8a-9c53-11ee-95a5-87774f69a715. 2023-12-16T21:28:31.845 INFO:journalctl@ceph.osd.1.smithi052.stdout:Dec 16 21:28:31 smithi052 systemd[1]: ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@osd.1.service: Consumed 20.719s CPU time. 2023-12-16T21:28:32.228 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:32 smithi052 systemd[1]: Stopping Ceph osd.2 for cf043a8a-9c53-11ee-95a5-87774f69a715... 2023-12-16T21:28:32.229 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:32.226+0000 7f097058b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-12-16T21:28:32.503 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:32.226+0000 7f097058b700 -1 osd.2 43 *** Got signal Terminated *** 2023-12-16T21:28:32.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:32.226+0000 7f097058b700 -1 osd.2 43 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T21:28:32.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:32.380+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:32.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:32.380+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:32.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:32.380+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:32.504 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:32 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:32.381+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:33.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:33.388+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:33.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:33.388+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:33.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:33.388+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:33.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:33 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:33.388+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:34.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:34.378+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:34.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:34.378+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:34.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:34.378+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:34.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:34 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:34.378+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:35.618 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:35.344+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:35.619 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:35.344+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:35.619 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:35.344+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:35.619 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:35 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:35.344+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:36.753 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:36.366+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:36.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:36.366+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:36.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:36.366+0000 7f096e38c700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:22:39.513295+0000 (oldest deadline 2023-12-16T21:22:59.513295+0000) 2023-12-16T21:28:36.754 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:36 smithi052 ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2[114585]: 2023-12-16T21:28:36.366+0000 7f096e38c700 -1 osd.2 43 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.24307.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:28:37.991 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:37 smithi052 podman[139310]: 2023-12-16 21:28:37.726423921 +0000 UTC m=+5.568065025 container died a382edce4272ad01af773aa0dbcc7ca1148742a0ba5e7fdc8e1b0999a30ce9d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.31.3, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20231212, RELEASE=quincy-1f2f551) 2023-12-16T21:28:37.992 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:37 smithi052 podman[139310]: 2023-12-16 21:28:37.873717395 +0000 UTC m=+5.715358501 container cleanup a382edce4272ad01af773aa0dbcc7ca1148742a0ba5e7fdc8e1b0999a30ce9d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=quincy-1f2f551, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD) 2023-12-16T21:28:37.992 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:37 smithi052 bash[139310]: ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2 2023-12-16T21:28:38.253 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:37 smithi052 podman[139329]: 2023-12-16 21:28:37.990153907 +0000 UTC m=+0.260233064 container remove a382edce4272ad01af773aa0dbcc7ca1148742a0ba5e7fdc8e1b0999a30ce9d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2, RELEASE=quincy-1f2f551, io.buildah.version=1.31.3, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20231212, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2023-12-16T21:28:38.728 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:38 smithi052 podman[139420]: 2023-12-16 21:28:38.348436629 +0000 UTC m=+0.059147363 container create c1763e0df578cee05238ca0971ccbec1b3b9bdc7b61da6b6a27f43dfbc668002 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, RELEASE=quincy-1f2f551, io.buildah.version=1.31.3) 2023-12-16T21:28:38.729 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:38 smithi052 podman[139420]: 2023-12-16 21:28:38.308297163 +0000 UTC m=+0.019007905 image pull 60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5 quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112 2023-12-16T21:28:38.729 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:38 smithi052 podman[139420]: 2023-12-16 21:28:38.432640124 +0000 UTC m=+0.143350869 container init c1763e0df578cee05238ca0971ccbec1b3b9bdc7b61da6b6a27f43dfbc668002 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=quincy-1f2f551, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2023-12-16T21:28:38.729 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:38 smithi052 podman[139420]: 2023-12-16 21:28:38.438048383 +0000 UTC m=+0.148759124 container start c1763e0df578cee05238ca0971ccbec1b3b9bdc7b61da6b6a27f43dfbc668002 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2-deactivate, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.vendor=CentOS, RELEASE=quincy-1f2f551, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2023-12-16T21:28:38.729 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:38 smithi052 podman[139420]: 2023-12-16 21:28:38.481788173 +0000 UTC m=+0.192498921 container attach c1763e0df578cee05238ca0971ccbec1b3b9bdc7b61da6b6a27f43dfbc668002 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=quincy-1f2f551, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.31.3, org.label-schema.build-date=20231212) 2023-12-16T21:28:38.729 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:38 smithi052 podman[139420]: 2023-12-16 21:28:38.727210202 +0000 UTC m=+0.437920955 container died c1763e0df578cee05238ca0971ccbec1b3b9bdc7b61da6b6a27f43dfbc668002 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20231212, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.31.3, RELEASE=quincy-1f2f551, org.label-schema.name=CentOS Stream 8 Base Image) 2023-12-16T21:28:38.985 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:38 smithi052 podman[139445]: 2023-12-16 21:28:38.956754635 +0000 UTC m=+0.218835765 container remove c1763e0df578cee05238ca0971ccbec1b3b9bdc7b61da6b6a27f43dfbc668002 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-cf043a8a-9c53-11ee-95a5-87774f69a715-osd-2-deactivate, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.31.3, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=quincy-1f2f551, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2023-12-16T21:28:38.985 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:38 smithi052 systemd[1]: ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@osd.2.service: Deactivated successfully. 2023-12-16T21:28:38.985 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:38 smithi052 systemd[1]: Stopped Ceph osd.2 for cf043a8a-9c53-11ee-95a5-87774f69a715. 2023-12-16T21:28:38.985 INFO:journalctl@ceph.osd.2.smithi052.stdout:Dec 16 21:28:38 smithi052 systemd[1]: ceph-cf043a8a-9c53-11ee-95a5-87774f69a715@osd.2.service: Consumed 18.955s CPU time. 2023-12-16T21:28:41.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:28:41.245 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:28:44.309 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:28:53.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:28:53.311 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:28:56.369 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:29:08.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:29:08.372 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:29:11.441 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:29:14.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:29:14.443 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:29:14.513 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:29:14.514 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T21:29:14.514 INFO:tasks.cephadm:Teardown complete 2023-12-16T21:29:14.514 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1540, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1794, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1797, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1673, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 178, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 297, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2023-12-16T21:29:14.537 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-16T21:29:14.548 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2023-12-16T21:29:14.564 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-16T21:29:14.564 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-16T21:29:14.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:29:14.585 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:29:17.653 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:29:26.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:29:26.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:29:29.713 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:29:41.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:29:41.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:29:44.789 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:29:47.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:29:47.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:29:47.857 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:29:47.858 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T21:29:47.908 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2023-12-16T21:29:47.925 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-16T21:29:47.938 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-16T21:29:47.938 DEBUG:teuthology.orchestra.run.smithi052:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T21:29:47.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:29:47.946 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:29:47.961 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2023-12-16T21:29:47.966 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-16T21:29:47.966 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2023-12-16T21:29:47.966 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 47 -184us[ -184us] +/- 70ms 2023-12-16T21:29:47.966 INFO:teuthology.orchestra.run.smithi052.stdout:^* hv02.front.sepia.ceph.com 3 7 377 103 +139us[ +139us] +/- 50ms 2023-12-16T21:29:47.966 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 +95us[ +95us] +/- 72ms 2023-12-16T21:29:47.966 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:29:51.029 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:30:00.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:30:00.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:30:03.094 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:30:15.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:30:15.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:30:18.161 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:30:21.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:30:21.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:30:21.237 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:30:21.238 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T21:30:21.238 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2023-12-16T21:30:21.252 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-16T21:30:21.263 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-16T21:30:21.264 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-16T21:30:21.320 DEBUG:teuthology.orchestra.run.smithi052:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-16T21:30:21.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:30:21.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:30:24.529 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:30:33.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:30:33.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:30:36.593 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:30:48.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:30:48.596 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:30:51.669 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:30:54.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:30:54.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:30:54.737 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:30:54.738 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T21:30:54.738 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2023-12-16T21:30:54.745 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-16T21:30:54.758 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-16T21:30:54.806 INFO:teuthology.task.internal:Duration was 3305.973070 seconds 2023-12-16T21:30:54.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-16T21:30:54.823 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-16T21:30:54.824 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-16T21:30:54.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:30:54.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:30:54.887 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-16T21:30:57.905 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:31:06.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:31:06.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:31:09.973 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:31:21.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:31:21.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:31:25.045 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:31:28.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:31:28.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:31:28.113 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:31:28.114 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T21:31:28.114 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2023-12-16T21:31:28.129 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-16T21:31:28.142 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-16T21:31:28.142 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-16T21:31:28.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:31:28.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:31:31.250 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:31:40.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:31:40.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:31:43.313 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:31:55.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:31:55.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:31:58.385 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:32:01.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:32:01.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:32:01.457 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:32:01.458 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T21:32:01.458 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2023-12-16T21:32:01.473 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-16T21:32:01.490 DEBUG:teuthology.orchestra.run.smithi052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-16T21:32:01.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:32:01.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:32:01.532 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = core 2023-12-16T21:32:04.561 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:32:13.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:32:13.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:32:16.629 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:32:28.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:32:28.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:32:31.697 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:32:34.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:32:34.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:32:34.769 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:32:34.770 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T21:32:34.770 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2023-12-16T21:32:34.784 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-16T21:32:34.797 INFO:teuthology.task.internal:Transferring archived files... 2023-12-16T21:32:34.799 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494793/remote/smithi052 2023-12-16T21:32:34.799 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-16T21:32:34.850 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494793/remote/smithi150 2023-12-16T21:32:34.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:32:34.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:32:37.909 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:32:46.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:32:46.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:32:49.969 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:33:01.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:33:01.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:33:05.042 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:33:08.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:33:08.044 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:33:08.117 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:33:08.118 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T21:33:08.118 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2023-12-16T21:33:08.134 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-16T21:33:08.148 INFO:teuthology.task.internal:Not uploading archives. 2023-12-16T21:33:08.149 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-16T21:33:08.159 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-16T21:33:08.160 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-16T21:33:08.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:33:08.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:33:08.183 INFO:teuthology.orchestra.run.smithi052.stdout: 265985 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 16 21:29 /home/ubuntu/cephtest 2023-12-16T21:33:08.183 INFO:teuthology.orchestra.run.smithi052.stdout: 265986 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Dec 16 21:32 /home/ubuntu/cephtest/archive 2023-12-16T21:33:08.184 INFO:teuthology.orchestra.run.smithi052.stdout: 265988 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 16 20:35 /home/ubuntu/cephtest/archive/syslog 2023-12-16T21:33:08.184 INFO:teuthology.orchestra.run.smithi052.stdout: 265990 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Dec 16 20:35 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T21:33:08.184 INFO:teuthology.orchestra.run.smithi052.stdout: 265989 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Dec 16 20:35 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T21:33:08.184 INFO:teuthology.orchestra.run.smithi052.stdout: 404502 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 16 21:30 /home/ubuntu/cephtest/archive/audit 2023-12-16T21:33:08.184 INFO:teuthology.orchestra.run.smithi052.stdout: 404504 292 -rw------- 1 ubuntu ubuntu 297907 Dec 16 21:30 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-12-16T21:33:08.184 INFO:teuthology.orchestra.run.smithi052.stdout: 276453 360 -rwxr-xr-x 1 ubuntu ubuntu 367151 Dec 16 20:43 /home/ubuntu/cephtest/cephadm 2023-12-16T21:33:08.185 INFO:teuthology.orchestra.run.smithi052.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-16T21:33:11.221 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:33:20.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:33:20.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:33:23.281 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:33:35.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:33:35.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:33:38.357 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:33:41.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-12-16T21:33:41.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:33:41.425 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-12-16T21:33:41.426 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T21:33:41.426 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2023-12-16T21:33:41.443 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-16T21:33:41.457 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-16T21:33:41.495 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2023-12-16T21:33:41.495 INFO:teuthology.nuke:Checking targets against current locks 2023-12-16T21:33:41.528 DEBUG:teuthology.nuke:shortname: smithi052 2023-12-16T21:33:41.528 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T21:33:41.552 DEBUG:teuthology.nuke:shortname: smithi150 2023-12-16T21:33:41.552 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T21:33:41.556 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494793', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 20:27:43.670978', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNUXelHJu4JJ7UoAMmLBbKww0hCCLDrnPDr5Hv1tiv/I80DAhvcxbJ+NCL5jHkJ5c2RJJ8kU/QoE2YAW1j0nKkw='} 2023-12-16T21:33:41.574 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494793', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 20:27:43.672111', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAE8XanDXS8HV8HzoKZ8bTXBKiWHqinDwT6anLD6hVY9hEIaHPxyUni6WKqjgHjWB85SKNd0Re2EUyO+B5GPOrs='} 2023-12-16T21:33:41.603 INFO:teuthology.orchestra.console.smithi052:Power off 2023-12-16T21:33:41.604 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-16T21:33:41.632 DEBUG:teuthology.orchestra.console.smithi052:power off output: Chassis Power Control: Down/Off 2023-12-16T21:33:41.632 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T21:33:41.648 INFO:teuthology.orchestra.console.smithi150:Power off 2023-12-16T21:33:41.649 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-16T21:33:41.660 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2023-12-16T21:33:41.672 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2023-12-16T21:33:41.673 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T21:33:41.697 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-12-16T21:33:45.662 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T21:33:45.699 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T21:33:45.773 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2023-12-16T21:33:45.810 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-12-16T21:33:49.777 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T21:33:49.813 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T21:33:49.888 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is off 2023-12-16T21:33:49.924 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-12-16T21:33:49.989 INFO:teuthology.orchestra.console.smithi052:Power off completed 2023-12-16T21:33:53.928 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T21:33:54.039 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2023-12-16T21:33:54.139 INFO:teuthology.orchestra.console.smithi150:Power off completed 2023-12-16T21:33:54.245 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3305.973069667816 failure_reason: 'Command failed on smithi052 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf043a8a-9c53-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e4bfae22d6f94512a47a4bb12e1b73b7 status: fail success: false 2023-12-16T21:33:54.246 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-16T21:33:54.332 INFO:teuthology.run:FAIL